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Abstract 


Today’s modern world often affords individuals to form teams to work 
together towards shared goals and objectives. The need for tools to digi- 
tally support collaboration distributed in time and space increased over 
the past decades significantly with a recent sudden increase due to the 
Corona pandemic crisis. The research goal in the field of computer-sup- 
ported cooperative work (CSCW) has always been to design systems that 
facilitate collaboration based on the understanding of groups and social 
interaction - especially on how people coordinate their work. 

In the real-world coordination happens in a seemingly seamless and 
effortless way. However, the resulting mechanisms translated to digital 
systems often provide a clumsy and awkward experience as users lack the 
means for subtle and rich interaction beyond the spoken word. After 
numerous failures revealing system deficits and a large number of 
ethnographic studies, researchers identified awareness to become the 
support mechanism for effortless coordination in digital systems. Yet, 
instead of addressing the problem using appropriate methods and tools, 
researchers found themselves trapped in circular reformulations of 
concepts and evaluations of prototypes on the basis of disciplinary 
preferences ignoring the basic characteristics of the objects of interest. 
Even worse, the most basic design tensions stemming from a use-inspired 
perspective have not been resolved indicating a substantial problem with 
the evaluation of awareness and coordination support. Effortless 
coordination cannot be reached without being measured, thus not without 
an appropriate measurement approach. 

This thesis introduces an appropriate assessment method for the ef- 
forts related to awareness and coordination support in cooperative set- 
tings — the STANDARDIZED COORDINATION TASK ASSESSMENT (SCTA). Ap- 
plying a use-inspired basic research driven approach it creates and lever- 
ages an effort-based operationalization of the two constructs derived from 
literature and especially from a cognitive perspective. A highly automated 
and scalable framework delivers quantitative results to be used for hypoth- 
eses validations that allows a benchmark-based approximation of effort- 
less coordination. At the same time the method opens the door for a lot 
more use-inspired basic research to resolve many of the still open design 
tensions and challenges. 
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Zusammenfassung 


Die moderne Arbeitswelt von heute bringt es mit sich, dass immer mehr 
Menschen als Team im Rahmen gemeinsamer Ziele und Vorgaben zu- 
sammenarbeiten. Der Bedarf an Werkzeugen, die die digitale und damit 
die gleichzeitige räumlich und zeitlich verteilte Zusammenarbeit unter- 
stützen, ist über die letzten Jahre stark gestiegen und erfuhr sogar in der 
jüngsten Vergangenheit eine zusätzliche starke Steigerung aufgrund der 
Corona Krise. Das Forschungsziel im Bereich Computer-Supported 
Cooperative Work (CSCW) bestand schon immer darin, die Zusammen- 
arbeit auf Basis des Verständnisses von Gruppen und sozialer Interaktion 
im Rahmen von sozio-technischen Systemen zu gestalten. Ein Hauptau- 
genmerk liegt dabei immer auch auf dem Aspekt der Koordination. 

Im Gegensatz zur digitalen Welt geschieht Koordination in der realen 
scheinbar mit Leichtigkeit und ohne große Mühe. Die für die digitale Welt 
entworfenen Möglichkeiten der Koordinationsunterstützung sind jedoch 
meist behelfsmäßig und eher ungünstig gestaltet, weil die Mechanismen 
für eine subtilere und reichhaltigere Interaktion, jenseits des 
gesprochenen Wortes, fehlen. Nach zahlreichen Fehlschlägen und einer 
Großzahl an ethnographischen Studien identifizierten Forscher 
„Awareness“ als zentralen Unterstützungsmechanismus für die 
mühelose Koordination in digitalen Systemen. Statt jedoch den 
Sachverhalt direkt mit geeigneten Methoden und Werkzeugen zu 
adressieren, fanden sich Forscher nach einiger Zeit in einem Zyklus der 
Reformulierung von Konzepten und der Evaluierung von Prototypen auf 
Basis von disziplinären Präferenzen wieder. Gleichzeitig wurden die 
Eigenschaften der Untersuchungsgegenstände häufig nicht beachtet. 
Zusätzlich wurden nicht einmal die grundlegendsten Forschungsfragen 
aus einer anwendungsinspirierten Perspektive beantwortet, sodass alles 
auf ein sehr grundlegendes Problem bei der Evaluierung von Awareness 
und Koordination hindeutet. Das Ziel der mühelosen Koordination selbst 
kann nicht ohne Messung erreicht werden und daher nicht ohne eine 
geeignete Messmethode. 

Diese Arbeit stellt einen angemessenen Ansatz für die Bewertung von 
Aufwänden im Kontext der Koordination bei der digitalen Zusammenar- 
beit vor — das STANDARDIZED COORDINATION TASK ASSESSMENT (SCTA). 
Unter Anwendung eines anwendungsinspirierten Ansatzes zur 
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Grundlagenforschung wird eine aufwandsbasierte Operationalisierung 
der beiden Konstrukte, die zuvor auf Basis einer Literaturstudie insbeson- 
dere unter Berücksichtigung einer kognitiven Perspektive erarbeitet wur- 
den, vorgenommen. Ein hoch-automatisiertes und skalierbares Rahmen- 
werk liefert quantitative Ergebnisse für eine Hypothesenvalidierung, die 
eine sukzessive Annäherung an die mühelose Koordination durch ein 
Benchmarking erlaubt. Gleichzeitig eröffnet die Methode Möglichkeiten 
für weitere anwendungsinspirierte Grundlagenforschung mit der viele 
der derzeit existierenden Gestaltungsprobleme angegangen werden kön- 
nen. 
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1 Introduction 


Nowadays, collaboration is key. Only few people achieve major results in 
their daily work completely on their own. The principle of the division of 
labor (Smith, 1776) has been introduced centuries ago, resulting in many 
different approaches like professional specialization, division of work 
within a company and between companies, or the split of work among 
man and machine (Bücher, 1893). 


Figure 1-1. Collaborative work as a collocated team (Source: Pixabay, CC0). 


With the advent of modern information technology (IT), collaboration 
lost its restrictions as to work together at the same time and place (cf. Fig- 
ure 1-1). For instance, “more than 26 million Americans — about 16% of 
the total workforce — now work remotely at least part of the time (...) Be- 
tween 2005 and 2015, the number of U.S. employees who telecommuted 
increased by 115%” (Greenbaum, 2019, p.54). There are further ad- 
vantages and reasons for this trend besides the technical evolution: the 
hiring of distributed talent (not only the one closest to the office), hiring 
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of staff outside expensive metropolitan areas, a gain in flexibility, time 
savings, reduction of transportation and child-care cost as well as a better 
work-life balance. Disadvantages of remote work especially include fewer 
opportunities to talk directly (i.e., fact-to-face) or to engage in side talks 
and to network or interact with colleagues. 

The recent Corona pandemic crisis that has started in late 2019 gave 
the trend another major push towards remote work. For instance, by mid 
2020 more than 60,000 out of 87,000 employees at Deutsche Bank worked 
from home or from elsewhere outside of the bank’s facilities (Jones & 
Maisch, 2020) - a number many deemed impossible even shortly before 
the crisis. Large numbers of people, even those until that point unwilling, 
were basically forced into the situation of having to use applications for 
computer-supported cooperative work (CSCW) for their daily tasks. “We 
Live in Zoom Now”, headlined an article in THE NEW YORK TIMES in early 
2020 (Lorenz et al., 2020). 


Figure 1-2. Working in a distributed setting using ZOOM (Source: Zoom Inc.). 
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ZOOM (cf. Figure 1-2) is a videoconferencing and collaboration tool of- 
fered by Zoom Inc.!. In detail it facilitates collaboration by providing the 
following functionalities: 


Integrated high definition (HD) video and audio supporting meet- 
ings up to 1,000 participants 

Recordings and transcripts generated using artificial intelligence 
(Al) 

Screensharing and annotations 

Calendar support for event scheduling 

Team chat 

Encrypted sessions and role-based security features 


Its competitor named TEAMS (cf. Figure 1-3) by Microsoft Inc.? gained 
a large share of new users during this crisis as well. 


Figure 1-3. Microsoft Teams (Source: Microsoft Inc.) 


1 https://zoom.us/ 


7 https://www.microsoft.com 
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Starting in 2017 as a chat-based workspace as part of OFFICE 3653, it 
recently passed the mark of 44 million users (Spataro, 2020). 


“Over the last three years, thousands of organizations, small and large—including 93 of 
the Fortune 100—have discovered how TEAMS can be their hub for teamwork, helping 
them to stay connected and engaged. Industry leading organizations are rolling out 
TEAMS enterprise-wide. In fact, 20 customers have more than 100,000 employees actively 
using TEAMS, including Ernst & Young, SAP, Pfizer, and Continental AG, as well as 
Accenture, which has 440,000 employees actively using TEAMS” (Spataro, 2020). 


Similar to ZooM, TEAMS offers the following features for remote col- 
laboration (Microsoft, 2020): 
e Shared workspace and file storage 
e Direct messages and group chat as well as channels for structur- 
ing threaded conversations 
e Videoconferencing (instant as well as live events for invited peo- 
ple) including screensharing 
e Integration into Microsoft’s OFFICE 365, e.g., to share documents, 
spreadsheets, calendars and emails 
The third tool that has gained great popularity in this area (even before 
the crisis) is SLACK of Slack Inc.*. It appeared years before TEAMS and was 
conceived by Microsoft as its main competitor and source of ideas 
(Warren, 2017). Therefore, Microsoft developed very similar features like 
the one of threaded communication, mentioned in the feature list above. 
SLACK (cf. Figure 1-4) specifically allows the structuring of text-based in- 
stant messaging communication into threads (also referred to as chan- 
nels) besides direct calls or multiuser conferences. Stewart Butterfield, 
creator and founder of Slack Inc., describes his tool as “all your commu- 
nication in one place, instantly searchable and available where ever you 


” 


go” (Hamburger, 2014). It integrates with a number of popular 


3 https://www.office.com/ 
4 https://slack.com/ 
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applications (e.g., TWITTER”, GITHUB® or DROPBOX’) on a channel basis. 
Originally SLACK even started out to “kill email” altogether (ibid.). 


Acme Inc v #social-media * (0) 
Matt Brewer 221 %1 Track and coordinate social media 


= Allunreads JE Arcadio Buendia 11:52 am 
Great work today everyone! 


Lisa Dawson 11:55 A 
EE Really need to give some kudos to @zoe for helping out with the new influx of tweets yesterday. People 
are really, really excited about yesterday's announcement. 


Bi 41 Kı 8: 


Zoe Maxwell 11:56 AM 
©) No! It was my pleasure! Great to see the enthusiasm out there. 
31 Acme Team APP 12:45 PM 207 le 
EI Event starting in 15 minutes: 

= #social-media * Qi 

| Team Status Meeting ¥ 

F e Zoe Maxwell 11.5: 
Today from 1:00 PM to 1:30 PM EJ No! It was my pleasure! Great to see 
the enthus 


Harry Boone 12:58 PM 
Quick note: today @Liza will join our team sync to provide updates on the lat 
bring ‘em. See you all later... er, in 2 minutes @ 


Team Status Meeting 3 
Today from 1:00 PM to 1:30PM 


Harry Boone 1258 pm 


# accounting 
EM Lee Hao 12:58 Pm 
W Meeting notes from our sync with @Liza 


a design-crit 


# help-design ck 
= team sync to pre 


jedia-and-pr launch. if you have q 


a pee Notes See you all later.. er, in 2 minutes © 
Last edited just now 
EM Lee Hao 12:55 P) 


# design-team-sf MM Meeting notes from our sync with @Lisa 
© Zenith Marketing is in this channel a 


B 1/9 Meeting Notes 
edited Just now 


Figure 1-4. Slack mobile and desktop frontends (Source: Slack Inc.). 


However, working with digital means to overcome separation and to 
collaborate across time and space introduces a lot of new challenges. It 
starts with technical issues, bad connections and security problems (as 
with the recent “Zoombombing”, i.e., the unwanted intrusion into a vide- 
oconferencing call causing disruption (Lorenz, 2020)) up to the overall is- 
sue of disembodiment (Bellotti & Sellen, 1993; Boyd, 2002; Heath & Luff, 
1993; Kang, 2007; Tian, 2017): Though using beautiful colored icons, great 


5 https://twitter.com 
6 https://github.com 
7 https://www.dropbox.com/ 
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photos, high definition audio and video, i.e., the best that technology has 
to offer, the digital world only provides awkward and imperfect means for 
rich communication to express feelings, moods, attitudes or other social 
affordances. Opposed to the real world where people instinctively find 
ways to collaborate seamlessly and effortlessly, the digital world still needs 
to explicitly find additional means to support the social and cognitive act 
of collaboration as a whole - the centerpiece thereof being the support of 
effortless coordination in distributed settings. 

The remainder ofthis chapter will provide a deeper background on the 
scientific context of this thesis, introducing the area of computer-sup- 
ported cooperative work (CSCW) especially focusing on support functions 
to collaboration including their evaluation as part of the design of CSCW 
applications with the goal to achieve effortless coordination for the digital 
world. Further, this chapter provides the fundamental problem statement, 
hypotheses and research objectives of this thesis as well as an overview of 
its overall contribution before eventually outlining its structure. 


1.1 Setting the Scene 


This research falls within the field of computer-supported cooperative 
work (CSCW), a theme in the study of human-computer interaction 
(HCI). HCI itself is an area of research and practice that emerged in the 
early 1980s especially with the advent of graphical user interfaces and the 
following consumerization of computer equipment. It is the region of in- 
tersection between cognitive sciences, human factors and computer sci- 
ence (Carroll, 1997; Grudin, 2012; Myers, 1998). 

CSCW attempts to achieve a deep understanding of group work and 
other types of social interaction to develop adequate technical concepts 
and tools for social interaction (cf. (Baecker, 1993; Greenberg, 1991; 
Gross, 2013)). In an early attempt, Wilson (1991) defined CSCW as “a ge- 
neric term which combines the understanding of the way people work in 
groups with enabling technologies of computer networking, and associ- 
ated hardware, software, services and techniques” (Wilson, 1991, p.1). In 
more detail, CSCW researchers are interested in the behavioral aspects of 
group activity (e.g. (McGrath, 1984)), the study of group interaction in the 
wild (e.g. (Heath & Luff, 1991)), the conceptualization and development 
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of appropriate tools and their continuous evaluation and adaptation 
(Borghoff & Schlichter, 2000). The term CSCW itself was coined by Irene 
Greif at one of the first workshops in 1984 (Greif, 1988). The major differ- 
ence between HCI and CSCW lies in the fact, there is not only one hu- 
man-being involved but at least two or more trying to interact using their 
computers — or information technology (IT) devices in more general 
terms. Mixed systems of this sort which are to be supported technically, 
are also referred to as socio-technical systems (Trist & Bamforth, 1951). 
They typically consist of the four components that became the Leavitt 
Rhombus (Leavitt, 1958): humans, tasks, technologies, and organizations 
(cf. Figure 1-5). 


cs 
Technology 


Figure 1-5. Components of group work aka Leavitt Rhombus (Leavitt, 1958). 


Cooperative work itself, which is a central aspect of this thesis, differs 
in many ways with some forms requiring closer interaction than others. 
This can be expressed by the degree of required communication (Bair, 
1989; Borghoff & Schlichter, 2000, p.110) (cf. Figure 1-6). 
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Inform Coordinate Collaborate 


low Degree of communication high 


Figure 1-6. Inform, coordinate, collaborate, and cooperate indicating closer interaction from 


left to right (Bair, 1989). 


There are four stages which at the same time serve as an indicator for 
the degree of work coupling (Neale et al., 2004): 

e Inform: This is the general exchange of information (not only re- 
lated to collaboration) indicating loose coupling. 

e Coordinate: People coordinate the information flow and their ac- 
tivities, e.g., when using shared resources.® 

e Collaborate People work together towards the same goal, yet, di- 
rect interaction remains sporadic, as they fulfill distinct and sepa- 
rate sub-tasks. 

e Cooperate: People share common goals, a common plan, and data 
that supports coordination. The demand of direct interactions 
(e.g., fact-to-face) is especially high, as teams work on shared tasks 
ofthe same goal indicating tight coupling. 


It is important to note that each stage requires the ones to its left (cf. 
Figure 1-6), i.e., cooperation and collaboration require coordination which 
requires information or better communication. 

Progress in CSCW used to be driven by new means of technology or 
new tasks or task requirements (Borghoff & Schlichter, 2000). However, 
more recently CSCW appears to focus more on research that describes 
collaborative environments in practice instead of testing hypotheses or de- 
veloping novel systems (Wallace et al., 2017). Moreover, CSCW is not lim- 
ited to computers anymore but rather encompasses all kinds of infor- 
mation technology (IT) devices. That is why the Association for 


8 This actually represents only a limited view on coordination. A more precise picture will 
be elaborated later in section 3.2. 
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Computing Machinery’s (ACM) CSCW conference generally focuses on 
“research in the design and use of technologies that affect groups, organ- 
izations, and communities” (Bietz & Wiggins, 2020). Actually, all other 
parts of the acronym CSCW advanced as well over the past decades 
(Wallace et al., 2017). 


“The terms computer, support, cooperative, and work have all been transcended. CSCW 
encompasses collaboration that uses technologies we do not call computers, 
collaboration in which technology plays a central rather than a support role, uses involve 
conflict, competition, or coercion rather than cooperation, and studies of entertainment 
and play” (Grudin & Poltrock, 2020). 


The IT systems being developed in this area are commonly referred to 
as groupware. This term was first defined by Johnson-Lentz as “com- 
puter-based systems plus the social group processes” (Johnson-Lentz & 
Johnson-Lentz, 1982, p.47). This very generic conception gained more de- 
tail as Ellis et al. (1991) defined it as “computer-based systems that support 
groups of people engaged in a common task (or goal) and that provide an 
interface to a shared environment” (Ellis et al., 1991, p.40). Additionally, 
groupware systems are also commonly referred to as CSCW applications 
(Grudin, 1994a). Both terms are used synonymously throughout this the- 
sis. ZOOM, TEAMS, and SLACK (cf. section 1) are all modern-day examples 
of CSCW applications or groupware. 

Groupware or CSCW applications have been defined and classified in 
many ways. A prominent classification scheme is the Time-Space Taxon- 
omy (Johansen, 1988) (cf. Figure 1-7), which was extended in many ways 
later on (cf. (Ellis et al., 1991; Grudin, 1994a) or section 5.2). It allows the 
differentiation of systems along the axes of their distribution in time and 
space. 

Another classification relevant here is the 3C model (Teufel et al., 
1995). It organizes CSCW applications according to their implementation 
of support functions for group activities: Communication support, coor- 
dination support, and cooperation support (cf. Figure 1-8).? 


9 This follows the line of (Bair, 1989) renaming inform to communication, keeping 
coordination and grouping collaboration and cooperation. 
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Same Time Different Times 
Face-to-face Asynchronous 
Same Place interaction interaction 
Synchronous Asynchronous 
Different Places distributed distributed 
interaction interaction 


Figure 1-7. Time-Space Taxonomy (Johansen, 1988). 


Communication support 


conferencing 
systems 


System classes 
Communication 

Shared information spaces 
Workflow management 
Workgroup computing 


Group editors 


Planning 


Workflow systems 


management 
systems 


Session support 
systems 


Coordination support Cooperation support 


Figure 1-8. 3C model based on CSCW support functions (Teufel et al., 1995). 
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Support functions are required by group members in order to cooper- 
ate effectively and efficiently across time and space (and team members) 
and to counter the effects of disembodiment (cf. section 1). ZOOM, TEAMS 
and SLACK can be placed in the middle of the 3C model since they offer 
features for communication (videoconferencing and text messaging), co- 
operation (shared workspace and file sharing), and coordination (team cal- 
endar). Gross & Koch (2007) later expanded the classification of CSCW 
support functions from three to five different types (cf. Figure 1-9). 
Among these support functions are the concepts of coordination and 
awareness support. The term coordination is defined as “the harmonious 
functioning of parts for effective results” or “the process of organizing 
people or groups so that they work together properly and well” (Merriam- 
Webster, 2016c). Awareness in cooperative systems is the mutual 
knowledge of each other’s presence and activities that has a positive effect 
on coordination (Dourish & Bellotti, 1992). Both concepts appear to work 
seamlessly and effortlessly in the real world (Schmidt, 2002, 2011a). 


Team support 
Support support support support support 


CSCW Awareness Communication Coordination Community | 


Figure 1-9. Classification of CSCW support functions (Gross & Koch, 2007). 


The design and development of groupware and technologies that is 
based on the understanding of how groups cooperate (Bietz & Wiggins, 
2020) tries to translate the same effortless experience of coordination to 
the digital world (Gross, 2013). The design typically involves an evolution- 
ary process joined by users and/or entire teams in the lab as well as in the 
wild (Dourish, 1995; Schmidt, 2009; Tang, 1991). Evaluations are an es- 
sential and critical part of these development cycles. In CSCW, these 
should not only cover the user’s main task but also the support functions 
for coordination and awareness — an issue unique to CSCW. The success- 
ful translation of effortless coordination would be proven by an appropriate 
evaluation in the form of before-after or A/B testing (Sauro & Lewis, 
2012). 
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Computer-Supported Cooperative Work (CSCW) 


Groupware & Support Functions 


Design & Evaluation of Groupware & Support Functions 


Assessment of Coordination and 
Awareness Support 


Figure 1-10. Research context of this thesis. 


This is where the problem addressed by this thesis begins: the assess- 
ment of coordination and awareness support. Figure 1-10 summarizes the 
overall research context. 


1.2 Problem Statement and Research Hypothesis 


In general, a research problem in HCI and thus also in CSCW “is a stated 
lack of understanding about some phenomenon in human use of compu- 
ting, or stated inability to construct interactive technology to address that 
phenomenon for desired ends” (Oulasvirta & Hornbaek, 2016, p.4960). 
HCI research is conceived as a problem-solving activity altogether. Scien- 
tific progress in this area improves the solution of problems related to hu- 
man use of computing. This in turn leads again to new research problems 
(cf. Figure 1-11). All in all, Figure 1-11 describes the evolution of HCI (and 
also CSCW) research. The problem addressed by this thesis is that there 
is no appropriate evaluation or assessment method available to judge on 
the efforts involved with coordination in computer-supported cooperative 
settings. The effects are that CSCW researchers are not able to tell whether 
or not they reached their goal of effortless coordination and what caused 
it to be reached. This further implies that researchers currently do not 
know what causes more or less effort when users are engaged in an activ- 
ity requiring coordination. Thus, it is unknown what preferred solutions 
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are to typical design challenges - this implies that a lot of basic research 
is needed once a method becomes available. 


i Research problem 
Human use of computing Empirical 


Improvement Conceptual 
potential Constructive 
Significance 
Effectiveness 
Efficiency 
Improvement in , Transfer 
problem-solving capacity Solution - Confidence 


Figure 1-11. HCI research problems (Oulasvirta & Hornbaek, 2016). 


From this situation, the following research hypothesis can be derived: 
in order to reach effortless coordination in digital systems, coordination (and 
awareness) need to be made measurable and to be measured. 

This research involves a lot of literature, concepts and terms from cog- 
nitive sciences. It depends 

“on the development of viable constructs and principles to promote better understand- 

ing of human performance in complex systems (...) [It] puts forward theoretically based 

constructs and tests their generality through empirical studies in a wide variety of labor- 
atory tasks, simulators, ‘microworlds,’ and actual work domains. A desirable feature is 
that the constructs and/or principles are quantifiable in the form of either mathematical 

or computational models” (Parasuraman et al., 2008, pp. 140). 


The American Psychological Association (APA) defines constructs as 
an “exploratory model based on empirically verifiable and measurable 
events or processes — an empirical construct — or on processes inferred 
from data of this kind but not themselves directly observable - a hypothet- 
ical construct” (APA, 2020). Most of the constructs of cognitive psychol- 
ogy are hypothetical constructs. For the case of awareness and coordina- 
tion this thesis connects the dots from previous work (main and helpful 
sidetracks) learning about their attributes and characteristics to form 
these constructs. 
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The above-mentioned research hypothesis will be investigated and 
tested by the research activities as part of the following research objectives: 


Objective 1: build a construct for awareness (conceptual operation- 
alization). This objective will be met by reviewing existing litera- 
ture on the subject from the area of CSCW gathering typical char- 
acteristics and underlying processes. 

Objective 2: build a construct for coordination (conceptual opera- 
tionalization). This objective will be met by reviewing additional 
existing literature from the areas of CSCW, HCI and cognitive sci- 
ences. 

Objective 3: design a measurement approach to assess awareness, 
coordination and respective support systems. This objective will 
be met by suggesting a method and to demonstrate its applicabil- 
ity with reliable and valid results. 


All of the work in this thesis builds on the following assumptions, 
some of which will be explained in greater detail in the course of this the- 


The progress in CSCW and more specifically with support func- 
tions is directly linked to how these means are designed and eval- 
uated. 

CSCW design more specifically fails at the evaluation stage due to 
many reasons (e.g., preference-based measurement, application 
of inappropriate methods, lack of appropriate methods etc.). 
Awareness and coordination have not been sufficiently studied 
and explained by CSCW to reach their effortless state in digital 
systems. 

Both constructs are something mental and have to be studied as 
such. That is, appropriate constructs need to be developed, as it is 
done for instance in cognitive psychology. 

Awareness and coordination can be evaluated separately and even 
in early stages of the development cycle as their means of support 
are independent of a situated context. 

Awareness and coordination are even independent of a specific 
task as coordination activities depend on the coordination’s inter- 
dependency type. 
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1.3 Scope 


This thesis does not seek to create large amounts of design principles 
ready to be applied by practitioners and the industrial sector, though this 
might be the result ofthe application of the constructs and method devel- 
oped as part of this thesis. 

Another topic somewhat related but out-of-scope are context-aware 
applications (Dey, 2001; Gross & Specht, 2001; Schilit & Theimer, 1994). 
A system is considered “context-aware ifit uses context to provide relevant 
information and/or services to the user, where relevancy depends on the 
user’s task” (Dey, 2001, p.5). In the context of this thesis, awareness is a 
vital characteristic of a human-being, not of a system. However, this does 
not exclude context to be a key information as part of awareness and 
coordination support systems. 


1.4 Contribution 


According to Evans et al. (2014) there are four general areas of contribu- 
tions: 

1. Theory development 

2. Tangible solutions 

3. Innovative methods 

4. Policy extensions 


Circling back to the beginning of HCI research (cf. section1.2), its 
research problem-solving can be categorized into three subtypes (cf. Fig- 
ure 1-12): empirical, conceptual, and constructive. 
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Research problems in HCI 


le Wi 


Empirical Conceptual Constructive 
1. Unknown phenomena 1. Implausibility 1. No known solution 
2. Unknown factors 2. Inconsistency 2. Partial solution 
3. Unknown effects 3. Incompatibility 3. Inability to deploy or 
implement 


Figure 1-12. HCI contribution types (Oulasvirta & Hornbaek, 2016). 


While empirical and conceptual types stem from Laudan’s Theory of 
Scientific Growth (Laudan, 1978), the constructive type was added to HCI 
to represent engineering and design contributions. Further, “this typology 
is orthogonal to the well-known Pasteur’s Quadrant which constitutes an 
attempt to bridge the gap between applied and basic research by suggest- 
ing ‘use-inspired basic research’ as an acceptable type”(Oulasvirta & 
Hornbaek, 2016, p.4958). Pasteur's quadrant (Stokes, 1997) (cf. Figure 
1-13) is a classification of scientific research seeking a fundamental un- 
derstanding of scientific problems, while also having immediate use. It 
bridges the gap between "basic" and "applied" research as coined by Bush 
(1945). 
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Considerations of use? 


No Yes 
Yes Pure basic Use-inspired 
research basic research 
Quest for 
fundamental 
understanding? 

A Pure applied 

No research 


Figure 1-13. Quadrant model of scientific research: Pasteur's quadrant (Stokes, 1997). 


In terms of these categorizations above, this thesis’ contribution in- 
cludes an innovative method (cf. Evans et al., 2014) comprised of a con- 
ceptual and constructive part (cf. Oulasvirta & Hornbaek, 2016), all of 
which belongs to the quadrant of use-inspired basic research (cf. Stokes, 
1997) considering use while at the same time seeking a fundamental un- 
derstanding of awareness and coordination support. In more detail, this 
contribution encompasses the following: 


A construct for awareness in CSCW (conceptual operationaliza- 
tion) 

A construct for coordination in CSCW (conceptual operationali- 
zation) 

A construct for the relation of awareness and coordination 

A quantitative evaluation method assessing the quality of aware- 
ness and coordination and their support in cooperative settings. 
A demonstration of the applicability of this method, especially in 
the context of forgetting and disruption in cooperative settings. 
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1.5 Structure 


All of the above will be described and achieved using the following struc- 
ture (cf. Figure 1-14): 


Chapter 1 introduces the basics for the term awareness, offering 
definitions and characteristics on the topic to build an initial con- 
struct extending on the introduction of CSCW (cf. section 1.1) 
and leading to the problems stated above (cf. section1.2). This 
chapter basically presents the story of awareness and awareness 
support as it can be found in standard CSCW textbooks. The 
chapter concludes with a discourse on issues that become appar- 
ent along the way. 

Chapter 3 adds the not so well-known aspects of cognition and 
coordination to the picture. This chapter establishes the construct 
of coordination also highlighting the differences to awareness as 
well as describing their relationship. It also closes with a dis- 
course on some of the issues raised in this chapter. 

Chapter 1 continues taking a look at the design and evaluation of 
CSCW applications, also focusing on challenges and difficulties 
regarding the evaluation and assessment of groupware, especially 
for the cases of awareness and coordination. As the previous two 
chapters this one closes with a discourse as well. 

Chapter 1 introduces the (constructive) solution to the problem, 
the STANDARDIZED COORDINATION TASK ASSESSMENTS (SCTA). It 
describes the goals, construction, implementation, evolution, 
and its means of analysis. A small evaluation of the solution con- 
cludes this chapter. 

Chapter 1 extends on chapter 1 describing the method’s applica- 
tions and experiments including findings and results. 

Chapter 1 summarizes the work presented in terms of meeting 
the research objectives. It provides an overview of possible future 
work and next steps and eventually draws a conclusion on the 
achieved objectives. 


Figure 1-14 depicts an overview of the structure. 
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Figure 1-14. Thesis structure. 


2 Awareness & Awareness Support 


At a first glance, targeting effortless coordination and then starting with 
awareness and awareness support instead of topics focusing on coordina- 
tion appears a little odd. Yet, this reflects how history evolved in the area 
of CSCW. Though actually discussing helpful coordination support func- 
tions researchers soon adopted the notion of making group members 
aware of the activities of the others. This is where it took off. 


“Not surprisingly then, the concept of ‘awareness’ has come to play a central role in 
CSCW, and from the very beginning CSCW researchers have been exploring how com- 
puter-based technologies might facilitate some kind of ‘awareness’ among and between 
cooperating actors” (Schmidt, 2002, p.285). 


This chapter is the first step towards the announced construct of 
awareness (cf. section1.2). There are many ways to approach an introduc- 
tion of awareness (cf. (Gross, 2013; Gross et al., 2005; Prinz, 2001; 
Rittenbruch & McEwan, 2009; Schmidt, 2002)). This chapter basically pro- 
vides a brief wrap-up of the facts to be found in standard CSCW literature 
and publications that are considered to be commonly accepted CSCW re- 
search results. 

Overall, this chapter has three main goals: 

1. Present an overview of CSCW’s research on awareness. 

2. Create a basic understanding of what awareness is. This forms the 
basis for an appropriate operationalization required for the target 
construct. 

3. Outline potential flaws and shortcomings in the current state of 
research. This forms the basis for improvements or redirects ef- 
forts towards a proper solution. 


It starts with the term’s origin and concept, then continues with the 
predominant dichotomy of ethnography and technology represented by 
studies, group characteristics and their tasks, as well as frameworks, and 
models. The chapter closes with a reflection on its contents outlining chal- 
lenges and criticism of what is known at this point to derive a construct 
before eventually closing with a summary. 
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2.1 Conceptual Aspects 


Simply checking a dictionary reveals that awareness is the state or quality 
of being aware, i.e., having or showing realization, perception, or 
knowledge of something happening or existing (Merriam-Webster, 
2016a). There it is also said to be synonymous to the term consciousness 
which rather refers to being awake to the surroundings and one’s own 
identity (Merriam-Webster, 2016b). However, the difference between the 
two terms becomes immediately obvious by using their antonyms: being 
unaware is truly something different than being unconscious. Conceiving 
it this way, it shows that consciousness is a prerequisite to become aware 
of something. 

In CSCW, the term awareness first appeared in the late 1980s and at 
the beginning of the 1990s: 


“More specifically, the early harvest of ethnographic field studies in CSCW (e.g., Harper 
etal., 1989b; Harper et al., 1989a; Heath and Luff, 1991) indicated that cooperating actors 
align and integrate their activities with those of their colleagues in a seemingly ‘seam- 
less’ manner, that is, without interrupting each other, for instance by asking, suggesting, 
requesting, ordering, reminding etc. others of this or that. As a placeholder for these 
elusive practices of taking heed of what is going on in the setting which seem to play a 
key role in cooperative work, the term ‘awareness’ was soon adopted” (Schmidt, 2002, 
p.285). 


Henceforth, awareness became one of the central “hot” topics of 
CSCW research starting in the early days with researchers exploring ways 
on how to support awareness using audio and video equipment as well as 
computer-based technologies. 

The first and most prominent definition of awareness, cited exten- 
sively in CSCW literature, stems from Dourish and Bellotti (1992): 
“Awareness is an understanding of the activities of others, which provides 
a context for your activity” (p.107). Yet, it was not the only definition 
around that time. Beaudouin-Lafon & Karsenty (1992) defined awareness 
in a similar way: „By this we mean that each user should be aware of what 
the others are doing, to facilitate coordination” (p.171). Over the years the 
term’s precision increased. Sohlenkamp (1998) extracts three common 
characteristics from the above and other sources: “They all imply that 
awareness is a state of mind of a user (‘understanding’, ‘knowledge’, ‘be 
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aware of), that it involves the activities of others, and that it provides a 
backdrop (‘context’) for own activities” (p.41). Yet, despite allthe attention, 
no clear overall picture of awareness has yet emerged from the CSCW re- 
search community until this day (Gutwin & Greenberg, 2002; Schmidt, 
2016). 

As stated in the beginning of this chapter, awareness and its charac- 
teristics can be seen from various perspectives making it hard to grasp in 
its entirety: “Awareness, like attention, is one of the tricky and dangerous 
terms in psychology, easily leading to circular argumentations” (Pedersen 
& Sokoler, 1997, p.51) and diverse definition approaches. Schmidt (2002) 
recounts it this way: 


“The very word ‘awareness’ is one of those highly elastic English words that can be used 
to mean a host of different things. Depending on the context it may mean anything from 
consciousness or knowledge to attention or sentience, and from sensitivity or 
apperception to acquaintance or recollection” (p.287). 


Research fields dealing with awareness range from anthropology and 
philosophy to human factors and psychology, each with a more or less 
different emphasis on certain details — yet all summarized under the very 
same term. Philosophers like Wolff depict awareness as the “conception 
of things” (Jacobs, 1973, p. 233). Yet, this is just one of its notions in the 
field of philosophy. For instance, the French philosopher Rene Descartes 
(1596 - 1650) describes awareness on the one hand as the knowledge of 
knowing, thinking, and imagining which corresponds to an unrestrained 
and abstract point of view (Descartes, 1637). In the same publication, he 
also refers to it as the knowledge of one’s own inner mental states — a very 
restricted perspective demarcating the individual from its environment or 
the mind from matter thus fitting the classic cartesian dualism (Descartes, 
1637). While the term’s first version appears to be very abstract and com- 
prehensive, the second is usually labeled “self-awareness” also due to its 
concrete notion closely related to Descartes’ prominent statement “Je 
pense donc je suis” (Descartes, 1637) later translated into Latin as “Cogito 
ergo sum” — “I think, therefore I am”. 

However, this actually has only little to do with what is being described 
in the context of computer-supported cooperative work, eventually leading 
to the extraneous use of prefixes, to discriminate the term from the above 
by focusing on a particular function or characteristic (e.g., a part of a 
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specific context). On the one hand, an overall encompassing and/or su- 
perordinate concept does not exist from which an appropriate sub-termi- 
nology can be deducted to be used in different research areas. On the other 
hand, this might exactly be what the term awareness is, as Sohlenkamp 
(1998) points out: “The term ‘awareness’ literally implies a broader mean- 
ing than actually intended by most related publications in CSCW litera- 
ture. What is usually meant is ‘group awareness’ or ‘cooperation aware- 
ness’, being knowledge about the state of a cooperative effort of a group 
of people” (p.40). 


2.1.1 Contents & Types 


Currently in CSCW, the usage of awareness with prefixes (ie., “xyz”- 
awareness) or as part of compounds adds its particular notion, thus also 
an idea of its content (e.g., self-awareness, presence awareness, social 
awareness, workspace awareness etc.). Doing so allows researchers to 
avoid two challenges: 1) to define an abstract term that fits all research as 
a common ground and 2) to integrate the own research with that of others 
as they marked theirs as something different. However, exactly this pro- 
cedure eventually leads to the fragmentation of research to be described 
(cf. section 2.4.5). For a first glance, Table 2-1 provides a sample of com- 
monly used awareness types in CSCW. 


Table 2-1. Sample of awareness types found in CSCW literature. 


Type Description 

Availability awareness Involves knowing who is around, whether they are busy or 
not (Jang et al., 2000). 

Activity awareness Reveals what other team members are doing (Jang et al., 
2000). 

Background awareness Provides “an overview of who was around and what was hap- 


pening (and afforded the possibility of joining in)” (Bly et 
al., 1993; Rittenbruch & McEwan, 2009). 


Collection awareness Combines the aspects of document and people awareness, 
explained below (Cohen et al., 2000). 
Contextual awareness Tackles the questions what users should be made aware of 


and how they should be made aware of it (Liechti, 2000). The 
user’s current context demands for certain pieces of infor- 
mation while ignoring others. 
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Conversational 
ness 


aware- 


“People continually adjust their verbal behaviour in conver- 
sation, based on cues picked up from their conversational 
partner (...) These cues provide a sense of awareness of what 
is happening in the conversation, awareness that helps us 
make adjustments and adaptations to keep things going 
smoothly” (Gutwin, 1997). 


Document awareness 


Refers to who sees the same document (e.g., a web page) at 
the same moment (Cohen et al., 2000). 


Environmental 
ness 


aware- 


Reveals all external information about activities and events 
in the agent’s environment as opposed to information about 
himself referred to as self-awareness (Boyd, 2002). 


General awareness 


“This is a pervasive experience, one of simply knowing who 
is around and something about what they are doing: that 
they are busy or free, meeting or alone, receptive to commu- 
nication or not” (Gaver, 1991; Bly et al., 1993). 


Group awareness 


It summarizes the knowledge about who is around and 
what they are doing, i.e., activity and presence awareness. 
(Dourish & Bellotti, 1992; Liechti, 2000). 


Group-structural aware- 
ness 


“Involves the knowledge about such things as people’s roles 
and responsibilities, their positions on an issue, their status, 
and the state of various group processes” (Greenberg et al., 
1996; Gutwin 1997). 


Informal awareness 


Represents the general knowledge of who is around in the 
work community (Greenberg et al., 1996). This type can also 
be found under the name people awareness (Cohen et al., 
2000) or availability awareness (Jang et al., 2000). 

It “involves knowing who’s currently around, whether 
they’re available or busy, and what sort of activity they’re en- 
gaged in” (Gutwin et al, 1996, p.205; Greenberg et al., 1996; 
Gutwin, 1997; Rittenbruch & McEwan, 2009) 

Knowledge of presence, activity and availability (p.30). 


Intentional awareness 


For instance, actively assess if a person is available 
(Pedersen & Sokoler, 1997). 


Mode awareness 


“Is the ability of a supervisor to track and to anticipate the 
behaviour of [mode-based] automated systems” (Gutwin, 
1997, p.20). 


Mutual awareness 


(Benford et al., 1994) 
(Pedersen & Sokoler, 1997) 


Organizational 
ness 


aware- 


Is the knowledge of how the group activity fits within a 
larger purpose of an organization (Gutwin, 1996). 


Passive awareness 


Awareness information that is gathered along the way 
(Dourish & Bellotti, 1992; Dourish & Bly, 1992) (cf. uninten- 
tional awareness). 


People awareness 


Describes knowing who is around (Cohen et al., 2000) (cf. 
presence awareness) 
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Peripheral awareness 


Rather denotes unobtrusive ways of presenting information, 
i.e., not requiring the focus of attention (Liechti, 2000). 
Weiser and Brown (1996) refer to means supporting periph- 
eral awareness as “calm technologies” which engage both 
the center and the periphery of our attention, and in fact 
move back and forth between the two (Gaver, 1992; Bly et 
al., 1993; Benford et al., 1994; Pedersen & Sokoler, 1997). 


Perspective awareness 


Refers to what team members are thinking and why (Jang et 
al., 2000). It considers personal background, training, and 
institutional contexts. Thus, it is a partial aspect of group 
structural awareness. 


Presence awareness 


Provides information on who is around and available 
(Dourish & Bellotti, 1992; Handel, 2001; Nardi et al., 2000) 


Process awareness 


Has an analogue characteristic to organizational awareness 
in that it refers to an understanding of one’s own work as 
part of a larger project or system (Jang et al., 2000). 


Self-awareness 


“Allows individuals to have a sense of who they are in rela- 
tion to society and culture” (Boyd, 2002, p.21), i.e., under- 
stand their representation and role during interactions be- 
sides their environmental awareness. 


Situation awareness 


“Situation awareness is the perception of the elements in 
the environment within a volume of time and space, the 
comprehension of their meaning, and the projection of their 
status in the near future” (Endsley, 1995, p.36). “SA is not 
general knowledge (long-term memory for facts, procedures 
or mental models), which has a relatively long 

time constant, being acquired (and forgotten) over periods 
of hours, days, and years. SA, on the other hand, generally 
applies to more rapidly evolving situations. Long-lasting 
knowledge supports but is distinct from SA” (Parasuraman 
et al., 2008, p.145). 


Social awareness 


Refers to the information that a person maintains about oth- 
ers in a social or conversational context (Greenberg et al., 
1996, p.30). 

“Social awareness includes information about the presence 
and activities of people in a shared environment” (Prinz, 
1999, p.392; Greenberg et al., 1996; Fogarty et al., 2004; 
Tollmar et al., 1996) 


Spatial awareness 


“Is a pilot’s understanding of her location in an airspace” 
(Gutwin, 1997, p.20). 


Task-oriented awareness 


“We consider task-oriented awareness as the awareness that 
is focused on activities performed to achieve a specific task. 
This kind of awareness can be promoted by change notifica- 
tions or information about the state of a certain document 
or shared workspace. It allows users to coordinate their ac- 
tivities on the shared object.” (Prinz, 1999, p.392). 


Unintentional awareness 


Maintained on others with no specific purpose (Pedersen & 
Sokoler, 1997). 
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Workspace awareness Is “the up-to-the-moment understanding of another per- 
son’s interaction with the shared workspace (...) workspace 
awareness is awareness of people and how they interact with 
the workplace, rather than just awareness of the workspace 
itself (...). [It] is limited to events happening in the work- 
place” (Gutwin and Greenberg, 2002, p.417). For Gutwin 
and Greenberg (2002) workspace awareness is a specializa- 
tion of situation awareness since in this case the situation is 
defined by people’s interactions within a shared workspace. 
Liechti (2000) describes workspace awareness as infor- 
mation about what artifacts are currently used by others, 
about their past changes, and who is around (Gutwin, 1997; 
Gutwin & Greenberg, 1999; Gutwin & Greenberg, 2002). 


This collection does not claim to be exhaustive. It rather seeks to 
demonstrate that there is a large number of definitions with many ofthem 
sounding quite similar or even being synonymous. The following relation- 
ships can be observed: 

1. There are types that comprise other types. For instance, presence 
awareness is part of group awareness, social awareness and work- 
space awareness. 

2. There are mutually exclusive types, e.g., self- and environmental 
awareness or presence and document awareness. 

3. There are types that differ in the degree of detail as with group 
awareness and presence awareness. 

4. There are types that differ in scope. Workspace awareness has an 
inherent spatial limitation to its workspace. Self-awareness is lim- 
ited to the respective self. 


Gutwin (1997) created the following depiction identifying further sub- 
type relationships (cf. Figure 2-1). He chose situation awareness (SA) to 
be the most generic of the awareness types. Therefore, it is shown at the 
top. SA originally stems from human factors and the research in the area 
of military aviation (Gutwin & Greenberg, 2002). 
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Situation 
Awareness 


Awareness of 
Spatial Awareness Mode Awareness others in 


collaboration 


Informal Conversational Structural Workspace 


Awareness Awareness Awareness Awareness 


Figure 2-1. Situation awareness and subtypes (Gutwin, 1997). 


Analyzing Table 2-1 in terms of what people are aware of reveals the 
following: 
e People (others) 
e Artifacts (e.g., documents, devices, shared resources) 
e = Events (i.e., what is happening in a certain context, activity of oth- 
ers, changes to documents and shared artefacts) 
e = Self (i-e., identity but also moods and feelings of oneself) 


Figure 2-2 employs the above list and depicts the links between aware- 
ness types and their scope. 

Leveraging Endsley’s (1995b) definition of situation awareness (cf. Ta- 
ble 2-1), all of it can be generalized into three major components each 
embedded in a particular context or environment (cf. Figure 2-3): 

e Elements (people and artifacts) 

e Events (affecting the subject and the elements) 

e Subject 
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Figure 2-2. Scope of awareness types. 


The subject, that is, the bearer of the awareness, needs to be added to 
Endsley’s definition. Awareness cannot stand for itself. It is always bound 
to individuals. “The term ‘awareness’ is only meaningful if it refers to a 
person’s awareness of something” (Schmidt, 2002, p.287). Thus, situation 
awareness is defined here as a subject’s perception of the elements in the 
environment within a volume of time and space, the comprehension of 
their meaning, and the projection of their status in the near future. 
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Figure 2-3. Elements and their relation in situation awareness. 


2.1.2 Process and Mental Model 


Most of the above-mentioned definitions of awareness obviously see it as 
something mental, dealing with perception, the processing and the result 
of what is being perceived. This leads to two ways awareness can be con- 
ceived (Gutwin & Greenberg, 2002, p.417): 
1. Awareness just as mental model (narrow sense) 
2. Awareness as mental model including the surrounding processes 
of perception and maintenance (wide sense) 


Craik (1943) and other cognitive psychologists refer to mental models 
in general as internalized representations or constructions of some 
aspects of the external world. In his theory of mental models, Johnson- 
Laird (1983) finds that they are not one-to-one imitations of the real world 
but much simpler structural analogues. Yet, the model’s similarity allows 
inferences and predictions for the real world to be made. 
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Integrating a mental model and its surrounding processes, Neisser 
(1976) developed the perception-action-cycle, a cognitive framework that 
shows the interdependencies of a person’s memory, perception and action 
(Adams et al., 1995). It describes interactions of a person with the envi- 
ronment and differs from standard linear information processing models 
(e.g., Input-Processing-Output (IPO) (Forouzan, 2017)) by recognizing 
that perception is influenced and directed by existing knowledge suggest- 
ing the involvement of the long-term memory (LTM) (Gutwin & 
Greenberg, 2002). 

In Neisser’s model (cf. Figure 2-4), subjects have a basic understand- 
ing (knowledge) about interacting with the environment. This directs 
their actions and activities (exploration) during which they gain further 
knowledge about their surroundings. The feedback of their actions is per- 
ceived and integrated into the existing mental models. On this basis, sub- 
jects might adapt their future actions (i.e., the ones of the next cycle). 


Environment 
(Objects) 


Modifies Samples 


Directs 


Knowledge Exploration 


(Schema) 


Figure 2-4. The perception-action-cycle (Neisser, 1976). 
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Based on the above the following characteristics of awareness can be 
summarized (Adams et al., 1995; Endsley, 1995): 

e It is the knowledge about the state of an environment bound in 
time and space. 

e The environment changes over time. Therefore, this knowledge 
needs to be maintained and updated. 

e People interact with and explore the environment. Through this 
interaction the update of awareness is accomplished. 

e Awareness is a secondary goal in the task. The primary goal is not 
simply to stay aware but to work on a cooperative task in the envi- 
ronment. 


Already Endsley’s definition of situation awareness (cf. Table 2-1) em- 
phasizes that awareness is achieved through three different stages or lev- 
els (Endsley, 1995) (cf. Table 2-2): 


Table 2-2. Situation awareness levels (Endsley, 1995). 


Phase Description 

Level 1 SA Involves perceiving the status, attributes, and dynamics of rel- 
evant elements of the environment 

Level 2 SA Comprehension of the situation based on Level 1 including an 
understanding of the significance of these elements 

Level 3 SA Projection based on Level 1 and 2 of the future actions of the 
elements in the environment near term 


2.1.3 Interrelations and Perspectives 


All types of awareness described in this thesis this far are unidirectional 
regarding the relationship of the subject and the surrounding elements, 
following the pattern “I know something about something”. This unidi- 
rectional type is also referred to as first level awareness (Oemig, 2004). 
Yet, real life is more complex, especially when it comes to groups of people 
and social interaction that relies on norms and conventions. “It is through 
such individual feelings of accountability that norms, rules and customs 
become effective mechanisms for supporting coherent social behavior” 
(Erickson & Kellogg, 2003, p.19). Accountability in turn relies on aware- 
ness but not the unidirectional type. It builds upon knowing that the other 
person knows that one knows. This can also be described as the awareness 
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about the awareness of another person and is referred to as second level 
awareness or social meta-awareness (Oemig, 2004). It describes the inter- 
relation ofthe awareness oftwo individuals (cf. Figure 2-5). Thus, it differs 
from the previous mutual awareness (cf. Table 2-1) in that both individu- 
als do not only know that they are around, for instance, monitoring each 
other without knowing of being monitored. They know from each other 
that each one ofthem knows that they are around. 


Figure 2-5. From mutual awareness a) to awareness about the awareness of others b). 


Social meta-awareness relies on people’s capability oftaking perspectives. 
“The ability to take the perspective of another is critical for effective social 
functioning (...) and is an important component of the ability to empathize 
with another” (Sheldon & Johnson, 1993, p.320). Based on taking perspec- 
tives, Sheldon and Johnson (1993) distinguish eight types of social aware- 
ness (cf. Table 2-3). 
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Table 2-3. Eight forms of social awareness (Sheldon & Johnson, 1993). 


Content Self as Target Other as Target 
Perspective of | Perspective of | Perspective of | Perspective of 
Self Other Self Other 
Experience Own experi- | Own experi- | Other’s experi- | Other’s experi- 
(covert) ence from per- | ence from per- | ence from per- | ence form per- 
spective of self | spective of | spective of self | spective of 
(privileged) other (non- | (non-privi- other _(privi- 
privileged) leged) leged) 
Appearance Own appear- | Own appear- | Other’s appear- | Other’s appear- 
(overt) ance from per- | ance from per- | ance from per- | ance from per- 
spective ofself | spective of | spective of self | spective of 
(non-privi- other (privi- | (privileged) other (non- 
leged) leged) privileged) 


Four out of eight forms are marked as “privileged” and four as “non-priv- 
ileged”. “Subjects indicated that they use the perspective of the other more 
often than their own only when it provides ‘privileged’ access to the rele- 
vant content of awareness — that is, when they think about the thoughts 
and feelings of the other or about their own appearance and behavior (...) 
The privileged perspective on covert experience is the experiencer’s, and 
the privileged perspective on overt appearance is an external observer’s 
perspective” (Sheldon & Johnson, 1993, p.320). In consequence, providing 
means for switching perspectives and objects facilitates the creation of 
empathy, self-and other-awareness effectively thus supporting accounta- 
bility. 

A similar path was picked up more recently when Tenenberg et al. 
(2016) introduced their notion of I-awareness and we-awareness. Previous 
concepts of awareness (cf. section 2.1.1) “are a first-person perspective 
that black-boxes the intentionality of others, focusing only on the actions, 
communication, and resources that are ‘publicly available’ (Robertson, 
2002), what we call I-awareness” (Tenenberg et al., 2016, p.236). This ba- 
sically refers to mutual awareness that was earlier described as unidirec- 
tional. In contrast, we-awareness corresponds to second level awareness. 
“Shared intentionality thus provides a basis for reconceptualizing aware- 
ness in CSCW research, building on and augmenting existing notions of 
individual intentionality. And it is just such a reconceptualization of 
awareness from ‘mutual awareness of something’ carried out seamlessly 
and effortlessly (Schmidt 2011), to a ‘shared awareness of something that 
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each recursively knows of the other’ (...). Our key move is in going from 
first-person singular to first-person plural, from I-awareness to we-aware- 
ness” (Tenenberg et al., 2016, p.236). 


2.2 Ethnographic Aspects 


Ethnography is the study and systematic recording of human cultures 
(Merriam-Webster, 2019b). “As a method of data collection [it] entails ex- 
amining the behaviour of the participants in a certain specific social situ- 
ation and also understanding their interpretation of such behaviour” 
(Dewan, 2018, p.188). The following briefly takes a glimpse at the results 
of well-known ethnographic studies as well as insights on group tasks and 
the effects on the concept of awareness. 


2.2.1 Studies 


Introduced in the Studies in Ethnomethodology, Garfinkel (1967) substan- 
tially challenged sociology as a discipline. In CSCW, ethnographic studies 
gained greater popularity when Suchman (1987) published her thesis on 
Plans and Situated Actions. In her opinion, there are “two alternative views 
of purposeful action and shared understanding”: 

1. Cognitive science: views the organization and significance of ac- 
tion as derived of plans (to prescribe possible actions). These 
plans are triggered by recognizing intent, common knowledge 
and typical situations. 

2. Social science: treats plans as derivate from situated action, i.e., 
ad-hoc responses to the action of others. 


In the early days of CSCW many approaches to support cooperative 
work failed (Grudin, 1988; Lynne et al., 1990) due to a missing reflection 
on social interaction in groups and organizations (Galagher & Kraut, 
1990). “There seems to be a focus on technology for the sake of technol- 
ogy, without much thought about what people need” (Henninger, 1991, 
p.28). That is why in the following years ethnographic studies gained a 
strong focus. For instance, Heath and Luff (1991) wanted to learn more 
about the subtleties of group work and focused on the Line Control rooms 
of the London Underground, a complex multimedia environment in 
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transition where they observed the informal work practices and proce- 
dures whereby personnel systematically communicate information and 
coordinate a disparate collection of tasks and activities. The underground 
employees developed special mechanisms of “monitor & display” where 
they overhear their colleagues and provide certain information them- 
selves, e.g., by talking out loud. 

Another study (Bentley et al., 1992) was conducted at the London Air 
Traffic Centre where radar controllers and assistants implicitly coordi- 
nated their cooperative work. Also this study demonstrated that coopera- 
tive work involved careful attention to ongoing events and actions on the 
one hand and the directed disclosure of important information to others 
on the other (Gross, 2013). Interestingly enough, neither study used the 
term “awareness” in conjunction with these mechanisms. 


2.2.2 Group Tasks and Group Dynamics 


The activities in the context of collaboration happen rather implicitly and 
invisibly. Ethnographic studies showed “that the effort of displaying and 
monitoring awareness information should be low enough so it can hap- 
pen in the background and does not interfere with the other activities of 
the actors” (Gross, 2013, p.430). This implies multiple tasks people are 
involved in during collaboration that might interfere with one another. 
Gutwin and Greenberg (2002) refer to these as domain and collaboration 
tasks (cf. Figure 2-6). 


Collaboration 


S Oo" 


+ 


Domain tasks 


Domain tasks Domain tasks 


Figure 2-6. Domain and collaboration tasks (Gutwin & Greenberg, 2002). 
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However, other authors use different terms for these tasks. Besides 
domain and collaboration task, for instance, Pinelle and Gutwin (2002) 
speak of the following two “distinct kinds of work: taskwork (the actions 
that must occur to complete the task) and teamwork (the actions that group 
members must carry out in order to complete a task as a group)” (p.456). 
Prinz (1999) distinguishes two types of awareness related to different 
tasks, namely between task-oriented awareness and social awareness (the 
one being about artefacts the other about people). 

These different tasks stem from the fact that not only one individual 
works on the task but multiple. Groups are “bounded, structured entities 
that emerge from the purposive, interdependent actions of individuals” 
(McGrath et al., 2000, p.95). However, “groups are not the same as teams. 
Groups have task structures with limited role differentiation, and perfor- 
mance depends largely on individual efforts. Teams, on the other hand, 
have members with specialized roles, and the team works together to ac- 
complish common goals” (Hare, 1992; Neale et al., 2004) (cf. Figure 2-7). 


Figure 2-7. Team or group? (Source: Pixabay, CCO) 


38 | Awareness & Awareness Support 


The interdependence of tasks and their coordination are defining char- 
acteristics of a team. 

McGrath (1984) is the one cited most when it comes to group tasks. 
His task typology goes far beyond the two above-mentioned task types. He 
organizes them into four quadrants that are subdivided into specific task 
types. In its entirety it is referred to as the Group Task Circumplex 
(McGrath, 1984) (cf. Figure 2-8). McGrath (1984) states that group inter- 
action “not only takes place somewhere, it involves the group doing some- 
thing. One very important aspect of all those settings just enumerated 
above is the ‘task’” (p.14). Tasks always involve goals (e.g., generating 
ideas, executing tasks, problem solving etc.). 
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Figure 2-8. The Group Task Circumplex (McGrath, 1984). 
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The quadrants of the circumplex are defined as follows: 

e Generate: contains tasks for structuring actions (planning activi- 
ties) and for producing ideas (creativity tasks). 

e Choose: contains intellective tasks for finding solutions to a prob- 
lem (problem solving) and reaching a consensus over the solution 
(decision making). 

e Negotiate: involves making choices in conditions of intragroup 
conflict, caused by opposing interests, as in bargaining (mixed- 
motive tasks) or by users systematically using different preference 
structures or viewpoints (cognitive conflict tasks). 

e Execute: mostly physical behavior such as contests or battles de- 
ciding on winners and losers (competitive tasks) or performances 
that emphasize on coordinating manual tasks of multiple people 
(psycho-motor tasks). 


Gutwin and Greenberg (2002) label the quadrants “Generate” and “Ex- 
ecute” as primary tasks that belong to the domain tasks. This thesis refers 
to everything that is part of the domain tasks as primary task and to the 
knowledge about the domain task as primary task knowledge. The pri- 
mary task in the context of CSCW requires collaboration. Everything that 
relates to the collaboration task, i.e., tasks related to coordination and 
awareness are referred to as secondary task and the knowledge thereof as 
secondary task knowledge. The secondary task is required by the primary 
task in a collaborative setting. In section 2.1.2, Gutwin and Greenberg 
(2002) referred to awareness as a secondary goal. 

Not all tasks of the Group Task Circumplex (cf. Figure 2-8) take place at 
the same time and not all of them are necessarily related to taskwork or 
teamwork. Some of them become relevant when it comes to the develop- 
ment of the team as a whole. They are not a secondary task to the domain 
task but triggered by the progress a team makes in terms of group dynam- 
ics. The term group dynamics itself goes back to Kurt Lewin (1890 — 1947), 
a German-American psychologist and later on director at the Center for 
Group Dynamics at the Massachusetts Institute of Technology (MIT). The 
term group dynamics describes how groups and individuals act and react 
to a changing environment (Lewin, 1939; Marrow, 1969). A first model to 
describe the stages of team development was developed by Tuckman 
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(1965) with the stages being forming, storming, norming and performing 
(Tuckman, 1965). This model was added an adjourning stage in 1977 
(Tuckman & Jensen, 1977). Team members first need to orient them- 
selves in the new structure, find their place, and setup standards for col- 
laboration before eventually performing the domain task at a high-perfor- 
mance level. After being done with the purpose of their formation, the 
team adjourns or reorganizes. Another more complex model of this kind 
is the Team Performance Model (TPM) (Drexler et al., 1998) (cf. Figure 2-9). 
It is more fine-grained and uses seven stages to describe the development 
of a team over time. In the “Orientation” stage team members find out 
why they are there and start finding a place within the group (as in the 
forming stage). During “Trust Building” team members try to find out 
about one another, about their plans and expectations. In the “Goal Clari- 
fication” stage the team decides on what to do. Once the goals are clear a 
way needs to be determined to achieve them in the “Decision Making” 
stage (as in the norming stage). Then follows the “Implementation” ad- 
dressing the sequence of work and who does what, when and where (i.e., 
coordination). During “High Performance” the plan is put into practice 
until the work is done and the team renews or adjourns. 

Models of group dynamics show that the tasks related to the team 
change over time. In consequence, the need for certain information 
changes as well. This corresponds to Neisser’s interaction with the envi- 
ronment (cf. section 2.1.2) with the information need changing the more 
is learned about it. That is, not only the domain tasks change over time, 
but also the information needs regarding a collaboration task. 
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Figure 2-9. Team Performance Model (Drexler et al., 1998). 
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2.2.3 Heed versus Attention 


One of Schmidt’s (2011) key insights was to note that the term “aware- 
ness” has been used in two distinct but at the same time incompatible 
ways in CSCW research, which caused the terminological confusion (cf. 
section 2.1.1) (Schmidt, 2011b; Tenenberg et al., 2016). The two ways are 
referred to as the attention and heed concept: 

e Awareness as a “attention” concept means being aware as being 
close to realizing or being conscious of or noticing. Already sec- 
tion 2.1 demonstrated the relation of awareness and conscious- 
ness with the latter being the prerequisite for being the former. 

e Awareness can be described as a “heed” concept when it refers to 
the mutual heeding that skilled actors perform in a real-world set- 
ting. The skillful technique of “monitor & display” (cf. section 
2.2.1) is the prime example for heeding which works in a seam- 
less and effortless manner. 


Heath et al. (2002) suggest “that awareness is not simply a ‘state of 
mind’ or a ‘cognitive ability’ but rather a feature of practical action which 
is systematically accomplished within developing course of everyday activ- 
ities” (p.318). That is why researchers like Schmidt suggest to learn more 
about how these skillful practices for heeding are obtained by team mem- 
bers: 

“Instead of searching for putative intermediate mental states, we should try to identify 

the strategies competent cooperating actors employ to heed what colleagues are doing 

etc. How do they discriminate significant states, possible states, problematic states etc.? 


What do they monitor for in the setting? What is ignored as irrelevant, what is taken into 
account” (Schmidt, 2011b, p.35). 


Obviously, ethnographers strongly favor the heed concept over the as- 
sumption of limited attentional resources that make it difficult to recog- 
nize that there might be forms of human activity in which an actor can 
attend to more than one thing while at the same time monitor the relevant 
activities of others. That is, Luff et al. (2008) critique conceptions of shared 
mental models, or the cognitive properties of individuals. As mentioned 
in the context of taking perspectives (cf. section 2.1.3), this is supposedly 
a reconceptualization of awareness from “mutual awareness of 
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something” carried out seamlessly and effortlessly (Schmidt, 2011b) to a 
shared awareness of something that each subject recursively knows ofthe 
others based on shared intentionality to allow socially recursive inference. 


2.3 Systems for Awareness Support 


After discussing the conceptual and ethnographic aspects, this section 
turns to the technological side. Human beings adapted to their natural 
environment over thousands of years. Yet, in the digital realm they start 
over and are reduced to windows and widgets: 


“We live in a complex world, filled with myriad objects, tools, toys, and people. Our lives 
are spent in diverse interaction with this environment. Yet, for the most part, our com- 
puting takes place sitting in front of, and staring at, a single glowing screen attached to 
an array of buttons and a mouse. Our tasks are assigned to homogeneous overlapping 
windows. From the isolation of our workstations, we try to interact with our surrounding 
environment, but the two worlds have little in common. How can we escape from the 
computer screen and bring these two worlds together?” (Wellner et al., 1993, p.24) 


Figure 2-10. Everything has to pass eventually the wire (Source: Pixabay, CC0). 
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In the digital realm, everything needs to be captured, encoded, medi- 
ated, decoded and displayed reducing the perceivable environment drasti- 
cally (cf. Figure 2-10). 


“The difference of environments (being immersed in an environment or just having a 
window into it) implies that our natural mechanisms for gathering awareness infor- 
mation do not work in the digital world (...) computational analogues are mostly slow 
and clumsy” (Greenberg et al., 1996, p.31). 


Before it becomes available, every piece of awareness and coordination 
support needs to be carefully designed and implemented and this can re- 
sult in many different systems and technological solutions. 

There are many ways to provide an overview of systems and prototypes 
supporting awareness. For instance, Rittenbruch and McEwan (2009) ap- 
ply a temporal structure identifying three different phases of system de- 
velopment (p.3): 

1. 1990-1994: Early explorations of awareness 

2. 1995-1999: Diversification and research prototypes 

3. 2000-now: Extended models and specialization 


Over the decades CSCW research generated many types of systems es- 
pecially to support awareness. The emphasis in the design of the early 
awareness support systems was largely on supporting peripheral monitor- 
ing, allowing people to see each other and their progress. Systems known 
as Media Spaces provided audio and video connections (i.e., high-fidelity 
awareness cues) to provide the distributed teams with a sense of who is 
around and their activities (Bly et al., 1993) (cf. Figure 2-11). 

“Unfortunately, however, the expected benefits from these technolo- 
gies never materialized” (Schmidt, 2002, p.285). Thereafter researchers 
introduced and worked on more abstract representations for the aware- 
ness of people and artifacts (Pedersen & Sokoler, 1997). 

Other researchers joint the development of Collaborative Virtual Envi- 
ronments (CVE) like MASSIVE (Benford et al., 1994; Greenhalgh & Benford, 
1995; Greenhalgh et al., 2000) where users completely immersed into 
their (virtual) working environment. However, “while impressive tech- 
nical progress has been made, it is still not clear if and how these technol- 
ogies could be used productively in cooperative work settings” (Schmidt, 
2002, p. 286). 
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Figure 2-11. Media Space (Bly et al., 1993). 


Another era that followed was the one of event notification systems. 
These are computational environments based on ‘event propagation 
mechanisms’ (Schmidt, 2002). They are used to provide information 
about the status of shared objects and the progress of collaborative tasks. 
Systems like GROUPDESK System (Fuchs et al., 1995), NESSIE (Prinz, 
1999), KHRONIKA (Lövstrand, 1991) or ELVIN (Segall & Arnold, 1997). 

Socially translucent systems (Erickson et al., 2002) are an example for the 
category of extended models and specializations. Their main feature is to 
make presence and social activity visible generating mutual awareness, 
which eventually reinforces accountability via taking perspectives and so- 
cial meta-awareness (cf. section 2.1.3). Thus, researchers try to transfer 
typical social phenomena to digital systems. 

Further specialization came from the area of dynamic awareness sup- 
port. The idea here is that not only the information presented to the user 
changes over time, e.g., presence awareness switching from “available” to 
“away” due to inactivity. It is the information set that changes altogether 
due to group dynamics (cf. section 2.2.2) or switching tasks (Oemig & 
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Gross, 2007). For instance, in instant messaging (IM) the presence aware- 
ness indicator is not of interest when conversing. In this case the applica- 
tion shows a typing indicator instead to coordinate the texting. However, 
the IM client returns to its former state once the conversation is over. 

There are about 30 years full of examples for CSCW applications sup- 
porting awareness, however, it is not the goal of this section to introduce 
each and every one of them.!° It is rather the goal to provide an overview 
over existing common components and terminology as well as operation- 
alizations introduced specifically by awareness models and frameworks 
that might help to build the required awareness construct. 


2.3.1 Terms and Components 


To build a system supporting awareness requires the term’s operationali- 
zation to translate it to the digital realm. As noted earlier (cf. section 2.1) 
awareness is something that resides in the mind of the user - the bearer 
of awareness. It is important to understand that the user is the one who is 
aware of something — not the system. That is the reason why two more 
terms need to be defined on the system side: 

1. Awareness information 

2. Awareness cues 


Awareness information “is all the information about existing objects 
and users within the system” (Antunes et al., 2001, p.33) or to stay with 
the definition of situation awareness (cf. section 2.1.1) it is all the infor- 
mation about the elements and events inside the subject’s environment. 
“All information” can be specified more precisely as the information an- 
swering questions (who, where, how, what, why), their origin (real world 
or digital world) and the point in time when it was created or captured 
(Sohlenkamp, 1998) (cf. Figure 2-12). 


10 For a more in-depth overview of these and corresponding details please refer to (Gross, 
2013; Rittenbruch & McEwan, 2009) or (Markopoulos et al., 2009). 
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Awareness & Awareness Support| 47 


N 

Who 6 

Where t 

How L 

What| O 
Why 


Figure 2-12. Three dimensions of awareness information (Sohlenkamp, 1998). 


Fuchs et al. (1995) distinguish awareness information in terms of syn- 
chronicity and coupling (cf. Table 2-4). This resembles the Time-Space 
Taxonomy (cf. Figure 1-7 in section 1.1), but leaves out the aspect of spatial 


distribution. 


Table 2-4. Synchronicity and coupling in terms of awareness information (Fuchs et al., 1995). 


Synchronous 


Asynchronous 


Coupled 


What is currently happening in 
the actual scope of work? 


What has changed in the actual 
scope of work since last access? 


Uncoupled 


What happens currently any- 
where else of importance? 


Anything of interest happened re- 
cently somewhere else? 
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In designing facilities that handle awareness information, four issues 
become extremely important (Donath et al., 1999): 
1. Data choice (what is the data to be visualized?) 
2. Data generation (where does the data come from?) 
3. Data mapping (how is the data presented?) 
4. The data’s impact (how does the interface affect the dynamics of 
the group?) 


On the other end, awareness cues are the user interface components 
to eventually trigger or causes the mental model in the user’s mind: 


“The cueing and construction of such a model can be supported by the way in which 
information is presented to a user: form and content of information determine which 
existing models are cued, and how new information is mapped onto existing or new 
models through the mechanism of procedural semantics. Applied to HCI, this means 
that designers would have to identify suitable existing knowledge to be cued, and present 
relevant information about the system in the context and form which directs the model- 
building process toward the intended mental model” (Sasse, 1997, p.40). 


In other words, “awareness cues are computer-mediated, real-time in- 
dicators of people’s undertakings, whereabouts, and intentions” 
(Oulasvirta, 2009, p.125). Facilities presenting awareness cues are also re- 
ferred to as awareness widgets (Gutwin & Roseman, 1996; Gutwin et al., 
1996) Awareness cues are not limited to the screen. They may be physical 
objects like the “Dangling String” created by Natalie Jeremijenko that vis- 
ualized the communication intensity of an ethernet cable (Weiser & 
Brown, 1996). There are high-fidelity cues (e.g., online audio and video 
connections as in Media Spaces (Bly et al., 1993) and low-fidelity cues us- 
ing icons as abstracted visualizations as with AROMA (Pedersen & Sokoler, 
1997). 


2.3.2 Models 


The models to be introduced here focus mainly on two aspects: 
1. Awareness information dissemination 
2. Determination of awareness levels 


The models that are responsible for the dissemination of awareness 
information are usually structured as event pipelines. That is why they are 
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often referred to as pipeline models (Fuchs, 1997; Rittenbruch & McEwan, 
2009; Sohlenkamp, 1998). Figure 2-13 depicts typical parts capturing the 
information by the sender and eventually reaching the receiver. 


1 ® Sender 


Provision of awareness 
information 


6 
Context 2 


Selection / 
Distribution 


Perception of 
awareness information 


Figure 2-13. Pipeline to distribute awareness information (Luczak & Wolf, 1999). 


Luczak & Wolf (1999) describe the following parts: 

The person who provides awareness information (sender) 
Specification and collection of awareness information 

Selection and distribution of awareness information 
Presentation of awareness information 

The person who receives (and perceives) awareness information 
(receiver) 

6. Context 


hes Si 


Sohlenkamp (1998) developed a more elaborate version of the aware- 
ness information pipeline (cf. Figure 2-14) adding additional filters 


Event Generation 
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allowing the sender to control the outgoing information (privacy filter) 
and the receiver the incoming information (filter controlling disruption). 


Global Filter: Individual Filter: Individual Filter: 

Allowed Outgoing Events Incoming Events 

Events 
= / Event History \ 

| | 
User Interface: nae i User Interface: 
Collection r Transport Presentation 
Mechanism 


Figure 2-14. Awareness information pipeline (Sohlenkamp, 1998). 


On the other end, there are models that help determine who becomes 
aware of what to what degree. For instance, the spatial model of interaction 
(Benford & Fahlen, 1993) is a model motivated by large virtual environ- 
ments with an underlying spatial metaphor. It can be applied to all CSCW 
applications that use a spatial metric meaning a way of measuring posi- 
tion and orientation. The space is filled with objects (or elements, cf. sec- 
tion 2.1.1) that may represent people, information or other artifacts. The 
users control their interaction based on levels of awareness between them. 
These levels are determined by the interplay of the two of the following 
concepts, focus and nimbus (Benford & Fahlen, 1993, p.112): 

e Medium: any interaction between objects occurs through a typical 

communication medium (e.g., audio, video, text etc.) 

e Focus: subspace, “the more an object is within your focus, the 

more aware you are of it” 

e Nimbus: subspace, “the more an object is within your nimbus, the 

more aware it is of you” 

e Awareness: “The level of awareness that object A has of object B 

in medium M is some function of A’s focus in M in relation to B’s 
nimbus in M” 


uondwnsuog Jusag 
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e Adapters: objects or tools to modify a subject’s aura, focus and 
nimbus, e.g., a microphone 

e Aura: a subspace which acts as an enabler of potential interaction, 
when two auras meet. An object may have different auras depend- 
ing on the medium (audio aura bigger than visual aura). 


Overall, the spatial model helps to facilitate interaction but is unspe- 
cific about what happens once a connection or link is established. How 
awareness levels are determined is show in Figure 2-15. 


a) Ais fully aware of B x 


A 
c) A unaware of B 


b) A is semi-aware of B 


Figure 2-15. The spatial model of interaction (Benford & Fahlen, 1993). 


The spatial model was applied in multiple systems, e.g., DIVE (Benford 
etal., 1995) and MAssIvE (Greenhalgh & Benford, 1995; Greenhalgh et al., 
2000). It also provided the basis for many other models some of which are 
described in the following. 

In his article Populating the Application (Rodden, 1996) describes a 
model of awareness based on the spatial model of interaction (Benford & 
Fahlen, 1993). “The model exploits the partitioning of space inherent 
within the spatial model to allow its application to non-spatial 
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applications” (ibid., p.87). Like the spatial model it is used to provide no- 
tions of presence, sharing and awareness. The model aimed to be gener- 
ally applicable across a variety of cooperative applications lacking a spatial 
metaphor. The model translates space into a simple “collection of objects 
shared by a number of users” (ibid., p.88). 

AETHER (Sandor et al., 1997) was the name of a generic awareness en- 
gine that also extended and reinterpreted the spatial model. Similar as 
Rodden (1996), Sandor et al. (1997) reinterpret the spatial model to be of 
“general utility beyond the domain of shared virtual environments” (ibid., 
p.222). “Our goal is to recognize awareness at a fundamental system level 
and to build other functions on top of it” (ibid., p.222). Therefore, they 
distinguish between an application space and an awareness space (cf. Fig- 
ure 2-16). CODESK (Tollmar et al., 1996) was the target system for their 
awareness engine. 


Applications 


Application Level 


Environment Level 


Awareness Level 


Awareness Engine 


Figure 2-16. CoDesk awareness engine AETHER (Sandor et al., 1997). 


The awareness engine itself can be found at the ground level of the 
system’s architecture. At the second level CODESK provides functionalities 
for shared file access, access control, versioning etc. “The top level is the 
application level where specific awareness information is collated and pre- 
sented to the user” (ibid., p.224). The application level may access the 
awareness engine directly or indirectly via the second level (cf. Figure 
2-16). 

Last but not least, the Model of Modulated Awareness (MoMA) (Simone 
& Bandini, 2002; Simone & Bandini, 1997) implements the so-called re- 
action-diffusion metaphor. Previous models focused on how awareness 
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information is produced and perceived but not on how it affects the be- 
havior ofthe objects. Simone and Bandini (1997) explain the components 
ofthe metaphor as follows: 


“Reaction refers to phenomena where two or more entities become in contact in some 
way and modify their state in consequence of this fact. Diffusion implies the existence 
ofa space where the involved entities are situated. Reaction-diffusion refers to situations 
in which the entities modify their state together with their spatial position (...) The math- 
ematical foundations of reaction-diffusion systems are due to A. Turing” (ibid., p.359). 


The described interaction with the environment appears to be closely 
related to Neisser’s (1976) perception-action cycle (cf. Figure 2-4 in section 
2.1.2). 


2.3.3 Frameworks 


In 2002, Gutwin and Greenberg published their workspace awareness 
framework (Gutwin & Greenberg, 2002). The goal was to develop a de- 
scriptive theory of awareness for aiding the iterative design of groupware. 
The framework itself was developed iteratively over a longer period of 
time. Their research hypothesis states “that helping people to stay aware 
in groupware workspaces will improve a groupware system’s usability” 
(p.412). It expands upon previous work (e.g., Neisser’s (1976) perception- 
action cycle (cf. section 2.1.2) and the theory related to situation awareness 
from human-factors (Endsley, 1995)) on the term awareness and focuses 
on only one specific awareness type (cf. Table 2-1 in section 2.1.1): work- 
space awareness. The scope was restricted to real-time distributed group- 
ware systems, that are systems that “allow people to work or play together 
at the same time, but from different places” (p.413). In part one of their 
framework, they defined the information that makes up workspace aware- 
ness (cf. Table 2-5 and Table 2-6). Here, they extend upon the characteris- 
tics of awareness information (cf. section 2.3.1) distinguishing elements 
related towards the present and elements related towards the past. 
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Table 2-5. Elements of workspace awareness relating to the present (Gutwin & Greenberg, 
2002). 


Category Element Specific questions 
Who Presence Is anyone in the workplace? 
Identity Who is participating? Who is that? 
Authorship Who is doing that? 
What Action What are they doing? 
Intention What goal is that action part of? 
Artifact What object are they working on? 
Where Location Where are they working? 
Gaze Where are they looking? 
View Where can they see? 
Reach Where can they reach? 


The elements towards the past are added two more categories, how 
and when. 


Table 2-6. Elements of workspace awareness relating to the past (Gutwin & Greenberg, 


2002). 


Category Element Specific questions 
How Action history How did that operation happen? 
Artifact history How did this artifact come to be in this state? 
When Event history When did that event happen? 
Who (past) Presence history Who was here, and when? 
Where (past) Location history Where has the person been? 
What (past) Action history What has a person been doing? 


Part two of their framework focuses on how awareness information is 
gathered. They identified three main sources and three corresponding 
mechanisms to gather it (pp.423): 

1. Bodies (source) and seeing and hearing activities, i.e., consequen- 

tial communication (mechanism) 

2. Artifacts (source) and feedthrough (mechanism, i.e., the feedback 

to the acting person that is used to inform others (Dix et al., 2004)) 

3. Conversation, gesture (source), and intentional communication 

(mechanism) 
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The third and last part of their framework describes how workspace 
awareness is used in collaboration. These activities are shown in Table 
2-7. 


Table 2-7. Usage of workspace awareness (Gutwin & Greenberg, 2002). 


Activity Benefit of workspace awareness 

Management of coupling Assist people in noticing and managing transitions be- 
tween individual and shared work. 

Simplification of commu- Allows people to the use of the workspace and artifacts as 

nication conversational props, including mechanisms of deixis, 
demonstrations, and visual evidence. 

Coordination of action Assists people in planning and executing low-level work- 
space actions to mesh seamlessly with others. 

Anticipation Allows people to predict others’ actions and activity at sev- 
eral time scales. 

Assistance Assists people in understanding the context where help is 
to be provided. 


In contrast to the spatial model of interaction which focuses on infor- 
mation selection and awareness levels the workspace awareness frame- 
work “is oriented towards small groups in medium-sized workspaces 
where it is more likely that participants are always interested in maintain- 
ing awareness of all the members of the group. Therefore, we see the fo- 
cus/nimbus model as a higher-level complement to our framework” 
(p.440). 

Further issues the workspace awareness framework does not consider 
are perspective taking and accountability (cf. section 2.1.3). However, 
these are regarded by the constraint-based awareness management 
framework or CoBRA (Gross & Oemig, 2006a) that extends the model of 
the awareness information pipeline (cf. section 2.3.2) while replacing the 
filters with more sophisticated constraints to support awareness especially 
in socially translucent systems (cf. section 2.3). 


WA Knowledge 


Who 
° Where r 
What Determine what to look for next 
whee * Selective attention 
* Expectations of future activity 
How 
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Environment 


Interpret perceptual information 

Interpretation is aided by: 

* Knowledge of the workspaces 
Knowledge of the task 

* Knowledge of the participants 

* Other WA knowledge 


Gather perceptual information 

* Consequential 
communication 

* Feedthrough 

* Verbal & non-verbal 
communication 


Knowledge Exploration 


* Explicit requests for WA information 


Uses of WA in collaboration 
* Simplification of communication 
* Coordination of actions and activities 
* Anticipation of events 
* Provision of assistance 
Management of coupling 


Figure 2-17. Overview of the workspace awareness framework (Gutwin & Greenberg, 2002). 


Constraints represent circumstances and conditions that have an im- 
pact on our interaction by supporting or inhibiting specific behavior 
(Bürgy & Garrett, 2002). These constraints can be grouped into the follow- 
ing categories (Gross & Oemig, 2006a): 

e Physical constraints (e.g., proximity, size, loudness etc.) 

e Social constraints (e.g., politeness, moral, decency, norms etc.) 

e Individual constraints (e.g., cognitive, logical, physiological etc.) 

e Legal constraints (e.g., rules and laws) 

e Organizational constraints (e.g., corporate culture) 

e Technical constraints (e.g., limitations due to the equipment used) 
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Some constraints can be violated (e.g., social constraints) while other 
cannot (e.g., physical). COBRA does not just use one constraint but a so- 
called constraint pattern consisting of multiple constraints describing ba- 
sically the context in which the interaction takes place (cf. Figure 2-18). 


ra 
5g 
u p 7 
N Constraint 2 
(5) 
Constraint 1 
° active À Server-push 
Originator a Sychnronous Gather Process Distribute à Recipient(s) 
passive f 


e Client-pull 
History g Asychnronous Retrieve Process Distribute q Requestor/ 
Recipient 


Flow of information ——————_> 


Figure 2-18. A constraint-based awareness management framework (Gross & Oemig, 


2006a). 


The spatial model (cf. section 2.3.2) can be implemented as physical 
constraint while perspective taking and accountability (cf. section 2.1.3) 
would serve as a social constraint. An implementation of the framework 
based on a social constraint was developed as part of the project 
PRIMIFaces (Gross & Oemig, 2006b) that focused on the selective dis- 
semination of awareness information in the context of instant messaging 
applications. 


2.4 Challenges, Reflections and Criticism 


As illustrated in the previous sections, research on awareness has been 
ongoing for quite some time. Many systems and prototypes have been de- 
signed and evaluated. However, there are still numerous issues that have 
not been resolved to this day. This section will deal with some of them 
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more specifically. They are all organized in short sections for easier refer- 
ence later. 


2.4.1 No Basic Research 


As elaborated in this chapter, one of the major goals of CSCW is to trans- 
late vital social mechanisms from the real world into technical concepts 
and means of support in the digital world. However, everything in the de- 
velopment of digital systems needs to be specified and implemented ex- 
plicitly. Due to this reduced environment, a lot of information is simply 
lost or not even captured in the first place. On the other end, people are 
easily overwhelmed by a lot less information (Gutwin & Greenberg, 2002). 
Maintaining awareness thus becomes exhausting and stressful due to 
non-existent or awkward and imperfect mechanisms supporting it 
(Gutwin & Greenberg, 2002, p.415): 
e Input and output devices only generate a fraction of the percep- 
tual information that is available in rich face-to-face situations. 
e A person’s interaction with a digital environment creates much 
less information than in a physical environment. 
e Many systems do not even present the limited awareness infor- 
mation that is available. 


As a consequence, designing awareness support does not only mean 
to display information properly but also to gather it in sufficient ways leav- 
ing researchers with a large amount of design challenges small and large 
that never had to be dealt with in the real world. This implies that a lot of 
basic research or rather use-inspired basic research (cf. section 1.4) is re- 
quired to catch up with reality as previously seamless and effortless rou- 
tines suddenly require effort in a digital setup (cf. section 2.2.3). At the 
same time, this is the worst starting point for CSCW’s goal of effortless 
coordination as researchers have to start at the very beginning. However, 
many researchers skipped ahead and left the basic part aside and devel- 
oped systems with features insufficiently backed causing many group- 
ware systems to fail altogether (Grudin, 1988, 1994b; Lynne et al., 1990). 
This was realized early on by other researchers complaining that “rather 
than looking at ‘fancy’ innovative functions for groupware systems, de- 
signers should be focusing on how to better solve the basic need” (Bullen 
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& Bennett, 1990; Henninger, 1991, p.294). Their overall assessment reads: 
“creeping featurism considered harmful to CSCW application” (ibid., 
p.28.). 


2.4.2 (Un-)Resolved Design Tensions 


As concluded in section 2.4.1, the research topic of awareness is full of 
design challenges. One of the most famous being the dual tradeoff be- 
tween awareness and privacy and awareness and disruption (Hudson & 
Smith, 1996). Awareness and supporting awareness as part of a support 
function of a cooperative application has a positive effect on the coopera- 
tion itself. However, there is also a downside or at least something con- 
sidered to be a tradeoff. After gaining some experience based on early 
CSCW applications, Hudson and Smith (1996) formulated the existence 
of the dual tradeoff between 1) awareness and privacy, and 2) awareness 
and disturbance. It basically states 


“the more information about oneself that leaves your work area, the more potential for 
awareness of you exists for colleagues. Unfortunately, this also represents the greatest 
potential for intrusion on your privacy. Similarly, the more information that is received 
about the activities of colleagues, the more potential awareness we have of them. At the 
same time, the more information we receive , the greater the chance that the information 
will become a disturbance to our normal work” (Hudson & Smith, 1996, p.248). 


The first is a sending problem, while the second one is a problem of 
receiving too much information. Thus, there are design challenges at var- 
ious levels from protecting data over to avoiding information overloading 
which is directly a result of the explicit effort to maintain awareness in the 
digital realm. 


2.4.3 The Unserved Mental Entity 


Awareness is described as mental model or knowledge or knowing some- 
thing (cf. section 2.1). It is even described to reside in the user’s mind. 
Therefore, it appears extremely odd that CSCW avoids cognitive aspects 
while the issue overall appears indicative of a cognitive approach. Instead 
researchers remain in the dichotomy of the two-sided discussion on 
awareness research: “the technology-oriented side where awareness is of- 
ten seen as technology providing information to users, and the 
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ethnographically-informed side where awareness is often seen as the out- 
come of an activity of the user” (Gross, 2013; Schmidt, 2011b, p.427). Es- 
pecially when seeking CSCW’s goal of effortless coordination an opera- 
tionalized cognitive construct is needed to measure efforts that suddenly 
appeared when translating real world mechanisms to digital systems (cf. 
section 2.4.1). Ethnographic studies are a useful tool but they will not help 
in this case. In the context of heed versus attention (cf. section 2.2.3) re- 
searchers even explicitly argued against cognitive approaches because of 
dealing with more than just a mental state or cognitive ability (Heath et 
al., 2002). “It does not make sense to conceive of ‘awareness’ as such, i.e., 
as a distinct (mental) entity. That is, the term ‘awareness’ is only mean- 
ingful if it refers to a person’s awareness of something” (Schmidt, 2002, 
p.287). Maybe, previous cognitive approaches in HCI and CSCW had their 
flaws but those can be fixed. They do not require to ignore cognitive sci- 
ence entirely. A dogmatic preference on research methods (Neale et al., 
2004) will not solve the design challenges at hand. 


2.4.4 Design Implications from Ethnographic Field Studies 


In CSCW, ethnography “has received serious attention as a method of in- 
forming system design, bringing a social dimension to the design process 
by focusing on how work is actually done rather than looking at these pro- 
cesses through some idealized organizational view” (Blythin et al., 1997, 
p-39). Researchers were quite certain that “empirical workplace studies, 
as much as they can provide specific design recommendations, are even 
more important to ‘contribute to a respecification of key concepts, like 
awareness, that are critical to an understanding of how technologies are 
used and deployed in everyday environments” (Luff et al., 2008, p.408). 
But the same way as ethnographers realized that “unfortunately, how- 
ever, the expected benefits from these technologies [supposedly support- 
ing awareness] never materialized” (Schmidt, 2002, p.285). The desire to 
provide design implications based on ethnographic studies did not mate- 
rialize, either. Plowman et al. (1995) wonder “why so few ethnographic 
studies result in specific design guidelines” (Plowman et al., 1995, p.312) 
as proclaimed. In their opinion these studies belong to very unique set- 
tings so that their results are hardly generalizable. At this point CSCW 
seems to ignore the general scientific procedures of hypothesis generation 
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and hypothesis validation, i.e., hypotheses being built on observations and 
qualitative research while hypothesis validation happens mostly on quan- 
titative data that needs to be measured. Design implications result from 
validated hypotheses whereas initial ideas for hypotheses may stem from 
ethnographic studies — in other words, it is not going to happen. 


2.4.5 The Problem with Awareness and the Big Hole 


“Understanding, defining, and operationalizing the many roles of aware- 
ness in collaboration is a key problem for the success of CSCW systems” 
(Convertino et al., 2004; Gutwin & Greenberg, 1996). But nearly three dec- 
ades after the initial findings, “nothing remotely similar to consensus 
about what the issue is and how to understand it has been achieved” 
(Schmidt & Randall, 2016, p.229). It appears to be even worse. Already in 
2002, Schmidt published an article in the Journal of Collaborative Com- 
puting and Work Practices (JCSCW) titled “The Problem with Awareness” 
stating: 
“The term ‘awareness’ is obviously found ambiguous and unsatisfactory. The term 
‘awareness’ of course refers to actors ‘taking heed of the context of their joint effort. But 
this is hardly a concise concept by any standard. CSCW researchers are obviously far 
from confident with using the term and thus often use the term in combination with 
different adjectives (...). The proliferation of adjectives is a clear indication that the term 
‘awareness’ is found to be equivocal, that researchers are aware that the term is being 


used in significantly different ways, and that it is in need of some qualification to be 
useful” (pp. 286). 


The term itself was already described as being highly elastic (cf. section 
2.1). “Interestingly enough, many authors discussing awareness issues 
rely on an intuitive notion of the term without giving any precise defini- 
tion. Where definitions are given, they usually are subtly different from 
those of other authors” (Sohlenkamp, 1998, p.40). 

For some reason researchers wish to distinguish themselves from oth- 
ers leading to the fragmentation of an entire research field as it was not 
only symptomatic for awareness alone. Schmidt (2009, p.223) diagnosed 
in his article Divided by a Common Acronym: On the Fragmentation of 
CSCW the following: 
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“the required continuity of the program, the ongoing development of concepts and 
frameworks, is replaced by restless reformulation of the research problem, slapdash 
changes of scope, unaccountable redefinitions of key concepts, etc. Under such circum- 


stances the progressive development of the conceptual foundation of technology is not 
possible.” 


Thus CSCW support functions are a prime example for the problem 
that surfaced eventually as the “big hole” in HCI (including CSCW) 
(Kostakos, 2015). In 2014 a bibliometric analysis was conducted (Liu et al., 
2014) spanning 20 years (1994-2014) on HCI’s flagship conference ACM 
CHI using co-word analysis (Callon et al., 1991; Callon et al., 1983). This 
type of study considers the keywords of papers, how these keywords ap- 
pear together on papers and how their relationships change over time. Co- 
word analysis, a content analysis technique, has been conducted for vari- 
ous disciplines (e.g., psychology, sociology etc.) over the years. It yields 
two values for each analyzed item: its density and its centrality. The den- 
sity (y-axis) reflects the internal cohesion of specific research theme. The 
centrality (x-axis) shows the strength of interaction between a specific re- 
search theme with others. Both are used to construct the strategic diagram 
(Callon et al., 1991) (cf. Figure 2-19). 


3 
a 
Quadrant Il Quadrant | 
Developed but isolated themes Motor themes 
“Ivory Tower“ “Mainstream“ 
> 
Centrality 
Quadrant Ill Quadrant IV 
Emerging or declining themes Basic and transversal themes 
“Chaos/Unstructured“ “Bandwagon“ 


Figure 2-19. Strategic diagram of co-word analysis (Callon et al., 1991). 
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The resulting quadrants are described as themes in addition to a catchy 

phrase or term: 

e Motor themes/”Mainstream” (quadrant I): research work here is 
internally coherent and central to the respective field. 

e Developed but isolated themes/”Ivory Tower” (quadrant II): inter- 
nally well-structured indicating that a group of researchers is ac- 
tively working on them. Yet, there are unimportant external ties 
leading to specialized/peripheral work. 

e Emerging or declining themes/”Chaos/Unstructured” (quadrant 
III): weakly developed with (yet) marginal interest from the re- 
search network. 

e Basic and transversal themes/” Bandwagon” (quadrant IV): weakly 
structured themes. Topics here are strongly linked to research in- 
terests but weakly linked together. There is potential of consider- 
able significance but work is under-developed as the research does 
not build on top of one another. 


The result for the ACM CHI conference, serving as an indicator for 
HCI in general, was not very pleasing. There were almost no items in the 
first quadrant. HCI obviously lacks motor themes and the “analysis 
showed that most research themes at CHI remain at the Bandwagon or 
Chaos quadrants. We simply roll from topic to topic, year after year, without 
developing any of them substantially” (Kostakos, 2015, p.50). An accumula- 
tion of knowledge obviously does not take place. Liu et al. (2014) presume 
the following reasons: 


“Due to the rapid pace of technology designed for humans, however, knowledge in HCI 
tends to be highly contextual instead of universal like in the field of biology or physics. 
So, we argue that by nature HCI research is like nomads chasing water and grasslands, 
making it challenging for the community to accumulate knowledge (...) We can hypoth- 
esize this to be a consequence of technological development and the need for novel in- 
teraction techniques” (p.3560). 


However, this statement rather sounds like an excuse instead of an 
explanation. On the other end, (Kostakos et al., 2015) make another point 
that potentially hits the mark: HCI research “completely overlooks the 
needs of fellow scientists and researchers. As a result, most research 
within HCI is not actually re-usable by other scientists” (p.3). This 
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prevents progress in research. Further, many researchers do not consider 
the “implication of design”, i.e., a mindset that places emphasis on re- 
usability for practitioners and the industrial sector. Research results need 
to be relevant to designing products — which implies that they are cur- 
rently not in every instance. All in all, this leads to the fragmentation of 
HCI (Liu et al., 2014, p.3560). 

The same co-word analysis of HCI also covers typical CSCW topics and 
places them in the fourth quadrant (“bandwagon”) as weakly structured 
themes. That means they are strongly linked to specific research interests 
throughout the network but are only weakly linked together. In other 
words, CSCW themes are under-developed yet transversal, with potential 
to be of considerable significance to the entire research (Liu et al., 2014, 
p.3554). 

Yet, ethnographic researchers see the problem in the concepts: “the 
difficulties in developing systems to support awareness do not simply de- 
rive from the limitations of technology, but rather from the ways in which 
we often characterize awareness and associated concepts such as mutual 
monitoring” (Heath et al., 2002, p.318). In Schmidt’s (2009) opinion, the 
CSCW community even ignores ethnographic results and mainly focus- 
ses on technology to support group work alone: 


“One example will suffice to indicate the level of fragmentation. Take the review article 
in the HCI Handbook, entitled ‘Groupware and computer-supported cooperative work’ 
(Olson and Olson, 2003). (...) In fact, this review of CSCW completely ignores the sub- 
stantial contribution of ethnographic or workplace studies to CSCW” (Schmidt, 2009, 
p.224). 


However, in terms of Kostakos (2015), there are hardly any design im- 
plications, resolved design tensions or re-usable concepts for practition- 
ers. As will be discussed later (cf. section 4.6.8), the real reasons appear to 
be something different. 


2.4.6 Awareness versus Coordination 


In section 2.4.3 awareness is described as something mental, yet cognitive 
science does not play a major role in CSCW as it has done in HCI. How- 
ever, this is not the only curiosity that strikes the eye. Another one is that 
in many definitions awareness is said to support or facilitate coordination 
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(cf. section 2.1). Even the overall goal refers to effortless coordination (cf. 
section1.2) and not effortless awareness. Early ethnographic studies did 
not even use the term awareness and rather spoke of “practices” like mon- 
itor and display to coordinate group work. The 3C model (Teufel et al., 
1995) named coordination as an important support function ofgroupware 
(cf. section 1.1), the awareness support function was added years later 
(Gross & Koch, 2007). Allin all, coordination appears to be the higher goal 
or the superior concept to awareness. This makes it even more surprising 
that a lot of research publications leave out coordination completely and 
focus solely on awareness. 

However, focusing more on coordination could have spared the re- 
search community a lot of trouble and confusion: as mentioned in section 
2.4.5, the problem with awareness started with the overloaded term itself 
which is also surrounded by large body of philosophical discussion blur- 
ring the sight towards the real thing. In comparison, the term coordina- 
tion appears rather technical, straight-forward and almost non-philosoph- 
ical overall. 


2.4.7 Context-Independent Awareness Support 


A fact mentioned earlier in this chapter that should not stay unnoticed is 
the one suggested by awareness models and engines in section 2.3.2: 
while the awareness of the subject itself is bound to a certain context (cf. 
section 2.1.1), the support system formed by awareness models and en- 
gines is not. Especially the AETHER model (Sandor et al., 1997) proposed 
a separation of application and awareness levels (cf. Figure 2-16 in section 
2.3.2). While the application belongs to the subject’s context, the aware- 
ness level contains the general means and mechanics!! to support aware- 
ness in many different applications and thus also contexts. 

A similar separation of context and mechanics can be found as part of 
the Clover model which also known as Clover architecture (Ellis & Wainer, 
1994; Laurillau & Nigay, 2002). “The Clover design model provides a high- 
level partitioning for reasoning about the collaborative services a group- 
ware application may support (...) a groupware application covers three 


11 These mechanics will be later referred to as the mechanics of collaboration (Gutwin & 
Greenberg, 2000). 
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kinds of services: production, communication and coordination” 
(Laurillau & Nigay, 2002, p.237): 
e The production space refers to the objects produced by group ac- 
tivity or to the objects shared by multiple users. 
e The communication space refers to person-to-person communi- 
cation such as e-mail, relay chat media space. 
e The coordination space covers activities dependencies including 
temporal relationships between the multi-user activities. It also re- 
fers to the relationships between actors and activities. 


There are two CSCW design issues attached to it: 

1. Not all groupware supports all of these spaces, e.g., media spaces 
(cf. section 2.3) lack a production space. 

2. Coordination spaces are sometimes implemented as separate sys- 
tems. 


That it can be done this way is proven by the following: The spatial 
model of interaction (Benford & Fahlen, 1993) is said to be used in wide 
range of different applications (Simone & Bandini, 1997, 2002). This 
would not have worked if they were dedicated to a unique context or setup. 
The topic of context-independence will be picked-up again later in section 
3.3.7 adding the perspective of coordination. For now, it boils down to: 
awareness is context-bound, the awareness support system is not! 


2.4.8 The Underrated Self-Awareness 


Reflecting on awareness and what has been said about it in this chapter 
reveals another underrated aspect in CSCW: self-awareness. Most defini- 
tions presented in Table 2-1 aimed at the knowledge of who is around and 
about the activity of others. Self-awareness has been described earlier in 
this thesis as the users’ “sense of who they are in relation to society and 
culture” (Boyd, 2002) adding the notion of identity to the picture among 
moods and feelings. 

In cooperative settings self-awareness also refers to the taskwork of 
oneself which is an important basis for coordination. Knowing one’s own 
part is the basis for making inferences on possible next options. As such, 
self-awareness has hardly been studied in CSCW leaving it as another 
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unresolved design challenge. Only in a few occasions it appeared, for in- 
stance, as part of Erickson’s (2003) six claims demanding a third-person 
point of view: 


“Although it might be argued that users do not need feedback on their own activity since 
they know what they’re doing, our experience is that this is quite important. People learn 
what elements of the social visualization mean by watching it over time, and , particu- 
larly, by seeing their own behavior reflected in it (...) Thus, a social visualization should 
show its users their own activity as others would see it” (Erickson, 2003, p.847). 


By revealing to the users how their activities are shown to others, en- 
ables perspective taking which is a basis for the principle of accountability 
based on the awareness of the awareness of others (cf. section 2.1.3). 


2.5 Summary 


In order to support coordination in digitally supported cooperative envi- 
ronments, CSCW researchers found and used the concept of awareness. 
“Understanding, defining, and operationalizing the many roles of aware- 
ness in collaboration is a key problem for the success of CSCW systems” 
(Convertino et al., 2004; Gutwin & Greenberg, 1996). Reflecting on what 
has been achieved in terms of awareness, reveals many conceptual ideas 
and technical implementations (Gross, 2013; Rittenbruch & McEwan, 
2009). After the initial failure of systems, researchers increasingly focused 
on ethnographic studies to better understand social interaction and user 
needs. Based on this knowledge they embarked in many directions with a 
constant reformulation of concepts and numerous unrelated technical 
prototypes each supporting a seemingly different type of awareness. 

Yet, the situation is not as bad as it seems. Especially, after going 
through conceptual aspects, models and frameworks provided a good 
sense of what awareness is and to start building a construct for it. 
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The key findings in this chapter are: 

e There is no coherent model or concept of awareness but when 
picking up the pieces it is possible to come up with a quite an 
extensive awareness construct. 

e The CSCW contribution to the concept of awareness is rather 
fuzzy. A clearer picture stems from human-factors and surround- 
ing psychological disciplines. Yet, these do not necessarily regard 
cooperative contexts. Situation awareness, a term stemming 
from human-factors, proved to be the most generic and compre- 
hensive in terms of definition and explanation. 

e Though being mental and for a starting point thus bound to the 
individual, awareness also develops a strong social impact, when 
introducing perspective-taking and meta-awareness. 

e Seemingly effortless coordination based on awareness can be dis- 
covered but not explained by ethnographic field studies. 

e Present awareness models appear to have a generic tendency 
splitting the awareness system from the application that fulfills a 
context-specific purpose. Typically, these models define compo- 
nents but no types of measurement of any sort. 

The summary of awareness attributes and processes gathered in this 
chapter can be found in Appendix D1. As a next step the cognitive com- 
ponents and processes need to be understood and specified in greater de- 
tail. Another part that has to be addressed is coordination and with it a 
clarification or extension on how all of this relates exactly to awareness (to 
be addressed in the next chapter). Last but not least, especially “seemingly 
effortless coordination” remains an open issue. The word seemingly indi- 
cates that this finding stems from observation. “Effortless” means (al- 
most) without effort. Diagnosing something as effortless requires an un- 
derstanding of where the efforts lie and how to measure them. 


3 Cognition, Coordination & Coordination Support 


As shown in the previous chapters, awareness has truly been a CSCW 
topic as such. Chapter 1 provides the main research results in terms 
awareness and discusses related challenges and problems. Especially the 
latter raise the following questions: 

1. If awareness is something mental (e.g.., a mental model), why 
have the cognitive details hardly been researched from a CSCW 
perspective? 

2. If awareness is often defined as “facilitating coordination” 
(Beaudouin-Lafon & Karsenty, 1992) or as “fundamental to coor- 
dination” (Dourish & Bellotti, 1992) (cf. section 2.1), why has 
there been only limited research in the terms’ relationship and 
meaning for one another? 


This chapter provides the missing details on cognition to reveal further 
aspects related to the human mind (addressing question 1) and on coor- 
dination to learn more about it as a construct and its relation to awareness 
(addressing question 2). Like the previous chapter, this one contains a dis- 
course on issues found through its sections followed by the chapter’s sum- 
mary where further characteristics of coordination are derived and gath- 
ered. 


3.1 Cognitive Aspects 


Section 2.1.2 showed that awareness can be described in a narrow and a 
wider sense: as mental model with and without the surrounding processes 
of perception and maintenance (Gutwin & Greenberg, 2002). HCI is 
known to be a field of applied cognitive engineering where insights from 
cognitive sciences (i.e., cognitive psychology, artificial intelligence, lin- 
guistics, cognitive anthropology, and the philosophy of the mind) inform 
the design of information technology for interaction (Norman, 1986; 
Wilson et al., 2013). It is “a type of applied Cognitive Science, trying to 
apply what is known from science to the design and construction of ma- 
chines” (Norman, 1986, p.31). 
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On the other end, cognitive sciences were not that prominent in 
CSCW. Standard text books on CSCW (Borghoff & Schlichter, 2000; Gross 
& Koch, 2007; Schwabe, Streitz, & Unland, 2001; Beaudouin-Lafon, 1999) 
do not feature a chapter on cognitive psychology or cognition in collabo- 
rative contexts. Their focus is directed right away towards groups and 
teams and their characteristics where sociological and ethnographic stud- 
ies are in the lead. Does that imply that there is no role for cognitive sci- 
ences in CSCW beyond the part already covered by HCI? Do cognitive 
aspects not change when moving from a single- to a multi-user setting (cf. 
section 2.2.2)? Do people stop using their perception and thoughts once 
in a collaborative context thus making cognitive sciences dispensable in 
CSCW? No. 

Cognitive psychology “is the branch of psychology devoted to the sci- 
entific study of the mind” (Braisby & Gellatly, 2005, p.2). It especially faces 
the challenge, that nearly everything it deals with is unseen or not directly 
observable (e.g., thoughts or memories). Therefore, cognitive psychology 
has to help itself with modelling the concepts to come up with predictions 
tested in appropriate studies. Card et al. (1983) state: “An applied psychol- 
ogy that is theory-based, in the sense of articulating a mechanism under- 
lying the observed phenomena, has advantages of insight and integration 
over a purely empirical approach” (Card et al., 1983, p.13) - as empirical 
approaches (like ethnographic studies, cf. section 2.1.3) rely on experience 
or observation without due regard for systems or theories (Merriam- 
Webster, 2019a). In the context of HCI (and probably also in the case of 
CSCW, as to be shown) the role of cognitive psychology is to specifically 
help with the evaluation of humans interacting with systems and technol- 
ogies and deriving design guidelines thereof. One example of the latter is 
the well-known ground rule of system development: “Recognition over re- 
call” (Budiu, 2014; Nielsen & Molich, 1990) or “Reduce short-term 
memory load” (Shneiderman & Plaisant, 2005). 

One step further down, cognition itself is the mental action or process 
of acquiring knowledge and understanding through thought, experience, 
and the senses or mental processes for short or just the product of these 
processes (Merriam-Webster, 2020). This sounds very familiar looking at 
the conception on awareness back at section 2.1.2. However, 
(Parasuraman et al., 2008) consider cognition which includes choice and 
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knowledge as the broader concept of both. The term cognition dates back 
to the 15" century, where it actually referred to “thinking and awareness” 
(Revlin, 2012). Back then, early cognitive models (cf. Figure 3-1) were de- 
veloped that even included gods (“deus”) and angels (“angelos”) besides 
more down-to-earth senses (“Auditus”, “Visus”, “Odorantus”, “Gustus”, 
“Tactus”) as well as man’s intellect and ratio. 


Figure 3-1. Cognitive model by Robert Fludd (1619). 
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The fairly similar definitions of cognition and awareness already indi- 
cate their close relationship, making awareness in cooperative settings 
even more eligible to be inspected from a cognitive point of view. 


3.1.1 Cognitive (Sub-)System 


Having cognitive psychology identified as an area of the “invisible”, pref- 
erably working with systems and theories, it is time to dig deeper into this 
world. “It is natural for an applied psychology of human-computer inter- 
action to be based theoretically on information-processing psychology, 
with the latter’s emphasis on mental mechanisms” (Card et al., 1983, 
p.13). By naming it information-processing psychology, Card et al. (1983) 
underline its close relationship to computer science with its typical sys- 
tems and components which will serve as a basic metaphor for their 
model: “A computer engineer describing an information-processing sys- 
tem at systems level (...) would talk in terms of memories and processors, 
their parameters and interconnections. (...) The human mind is also an 
information-processing-system, and a description in the same spirit can 
be given for it” (Card et al., 1983, p.24). 

Having this in mind, Card et al. (1983) proposed the well-known 
Model Human Processor (MHP) in their seminal book The Psychology of 
Human-Computer Interaction. In this model they suggest three interacting 
subsystems (cf. Figure 3-2): 

1. Perceptual subsystem: carries sensations of the physical world de- 
tected by the body’s sensory systems into internal representations 
of the mind. 

2. Cognitive subsystem: connects inputs from the perceptual system 
to the right outputs of the motor system. 

3. Motor subsystem: translates into action by activating patterns of 
voluntary muscles. 
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Figure 3-2. Model Human Processor (MHP) (Card et al., 1986; Card et al., 1983). 


Each subsystem has its own memory/storage and processor. The sys- 
tem as a whole may operate in different modes. For some tasks the human 
must act as a serial processor, others require parallel operation of all three 
subsystems. The memory entities are described by the following parame- 
ters (Card et al., 1983, p.25): 

e The storage capacity in items (u) 

e The decay time of an item (5) 

e The main code type (physical, acoustic, visual, semantic) (x) 


Whereas the most important characteristic for a processor is its cycle 
time (t). The cycle time for the cognitive processor (tc) is “shorter when 
greater effort is induced by increased task demands or information load; 
it also diminishes with practice” (Card et al., 1983, p.42). Card et al. named 
this the Variable Cognitive Processor Rate Principle. 

“The basic principle of operation of the Model Human Processor is the 
Recognize-Act Cycle of the Cognitive Processor (...) On each cycle of the 
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Cognitive Processor, the contents of Working Memory initiate actions as- 
sociatively linked to them in Long-Term Memory; these actions in turn 
modify the contents of Working memory” (Card et al., 1983, pp.26). This 
comes close to the previously described perception-action cycle (Neisser, 
1976) (cf. section 2.1.2). 

However, the model of the MHP is a general description not specific 
to the context of CSCW or even awareness. On the other end, not dealing 
with aspects of CSCW is one of its limitations. The MHP is based on psy- 
chological sciences thus leaving out social or organizational aspects rele- 
vant in CSCW and cooperative contexts. 

The following sections especially focus on the cognitive subsystem. It 
is comprised of the working memory, long-term memory, and the cogni- 
tive processor (cf. Figure 3-2). Card et al. (1983) characterize it as “funda- 
mentally parallel in its recognizing phase and fundamentally serial in its 
action phase. Thus, the cognitive system can be aware of many things, but 
cannot do more than one deliberate thing at a time” (p.43). 


3.1.2 Perception, Attention and Memory 


Like awareness, perception is again a term with two meanings, one of the 
process and the other the result of that process. Perceptions enter through 
the visual, auditory, olfactory, gustatory or tactile sensory systems and trig- 
ger responses starting in areas of the brain dedicated to each sense 
(Johnson, 2010) (cf. Figure 3-1). The MHP reduces its perceptual system 
to auditory and visual perception (cf. Figure 3-2) that are considered the 
“most important buffer memories” (Card et al., 1983, p.24). 

As to cope with this potentially large amount of information, the mind 
or more specifically the cognitive subsystem employs a filter mechanism 
named attention. “The human brain has multiple attention mechanisms, 
some voluntary and some involuntary” (Johnson, 2010, p.82). They focus 
our awareness on a very small subset of the perceptions and activated 
long-term memories while ignoring everything else, i.e., the opposite to 
attention is ignorance. From that perspective, awareness can be described 
as “our ability to maintain and constantly update a sense of our social and 
physical context. We do so in an apparently effortless manner and without 
being aware that we do so - at least until something happens that is out 
of order and makes us raise our level of consciousness” (Pedersen & 
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Sokoler, 1997, p.51). Humans shift their attention when needed. Taking 
(again) an information processing perspective, attention belongs to the 
memory management procedures that keeps the system from overloading 
as the human information processing system has a limited capacity 
(Baddeley, 1999; Chandler & Sweller, 1991). Attention is the mechanism 
that avoids that too much information enters the working memory. On 
the other end, there is another mechanism that purges the information 
from memory to release some capacity: forgetting (cf. section 3.1.3). 

Cognitive processes are continuously taking place in your mind and in 
the minds in the people around you (Sternberg, Mio, & Sternberg, 2011) 
(cf. Figure 3-3). 


Figure 3-3. The human mind - not the brain (Source: Pixabay, CC0). 


It has to be noted that cognitive psychology always refers to this overall 
structure as the mind and not the brain which is a term from neurobiology 
(Braisby & Gellatly, 2005). Other cognitive scientists refer to the mind as 
the software of the brain (Block, 1995). 
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“How does the mind work? Though this is, of course, the central question posed by 
cognitive science, one of the deepest insights of the last half-century is that this question 
does not have a single answer: there is no one way the mind works, because the mind is 
not one thing. Instead, the mind has parts, and the different parts of the mind operate 
in different ways: seeing a color works differently than planning a vacation, which works 
differently than understanding a sentence, moving a limb, remembering a fact, or feel- 
ing an emotion” (Firestone & Scholl, 2016, p.4). 


Besides the cognitive processor the Model Human Processor intro- 
duces the working memory and the long-term memory as part of the cog- 
nitive subsystem (cf. Figure 3-2). In general, the term memory is used in 
multiple ways to describe the following (Ritter et al., 2014, p.123): 

1. The mental function of retaining information about things (stim- 
uli, events, images, ideas etc.) when those things are no longer 
present. 

2. The hypothesized storage system of the mind where this infor- 
mation is stored 

3. The information itself that is stored. 


Structures for storage are typically called stores. The information in- 
side of these is referred to as memory (Sternberg etal., 2011, p.193). There 
are three common operations applied to stores by the processor (Baddeley, 
2002; Brown & Craik, 2000): 

1. Encoding: sensory data is transformed into a mental representa- 

tion 
Storage: information is kept encoded in memory 
3. Retrieval: pull out or use information stored in memory 


Over time, multiple major models of memory (McAfoose & Baune, 
2009; Murdock, 2003) were developed. William James (1970) distin- 
guished two major types: 

e Primary memory: temporary information 

e Secondary memory: permanent or at least for a longer time 


Atkinson and Shiffrin (1968) already used three store types. Their 
model became known as the traditional three store model (Sternberg et al., 
2011): 
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e Sensory store: capable of storing limited amounts of information 
for brief periods 

e Short-term memory (STM): capable of storing information some- 
what longer but limited in capacity 

e Long-term memory (LTM): capable of storing information for very 
long periods, large capacity 


When it comes to short-term stores, Baddeley’s working memory 
model is “probably the most widely used and accepted model today” 
(Sternberg et al., 2011, p.203). While Atkinson and Shiffrin (1968) sug- 
gested that the STM acts as a working memory responsible for a variety of 
control processes, Baddeley and Hitch (1974) explored and expanded their 
idea and concluded that STM is better regarded as a component of work- 
ing memory (Braisby & Gellatly, 2005, p.313). Baddeley and Hitch (1974) 
originally suggested three major components (cf. Figure 3-4): 

1. Visuo-spatial sketchpad: a system for maintaining and manipulat- 

ing visual images 

2. Phonological loop: an auditory store and rehearsal process 

3. Central executive: an attentional control similar to the cognitive 

processor of the MHP 


Later, a fourth component was added (Baddeley, 2000): The Episodic 
buffer, which is story based also drawing on information from the long- 
term memory. The central executive can be seen as the processor and the 
other components as distinct stores with a certain way of storing infor- 
mation. 


“The central executive of working memory is assumed to be a limited-capacity atten- 
tional system that controls the phonological loop and sketch pad, and relates them to 
long-term memory. The executive is almost certainly considerably more complex than 
either of the two slave systems, which makes it considerably harder to investigate” 
(Baddeley, 2013, p.66). 


Further, an attentional system implies volatility: it focuses the atten- 
tion on new information thus turning it away from some of what it was 
focusing on earlier. “If items in short-term memory don’t get combined 
or rehearsed, they are at risk of having the focus shifted away from them” 
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(Johnson, 2010, p.84). The content is easily lost as it endures only as long 
as we are paying attention to it (Revlin, 2012). 
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Figure 3-4. Working Memory Model (Baddeley, 2000; Baddeley & Hitch, 1974). 


From a computer science and programming perspective this sound 
like a garbage collection mechanism that reclaims the storage associated 
with an object when it becomes unreachable, i.e., when nobody refers to 
it any longer (Bloch, 2018). This is another instance where human cogni- 
tive characteristics are similar to concepts in computer science - starting 
out with the MHP earlier in this section. 

Already the MHP suggests a connection between awareness and the 
working memory (cf. Figure 3-2). The MHP and Baddeley’s theory ofthe 
working memory both follow three assumptions (Mayer & Moreno, 2003, 
p-45): 

1. Dual channel: Humans use separate information processing 
channels for verbal and visual material. It is a central theme of 
Paivio’s dual-coding theory (Paivio, 1986, 1991) (cf. Figure 3-5). 
Here, there are not only two channels but signals coming in both 
ways are encoded in an associative manner making them easier to 
remember compared to the encoding in a distinct manner. 

2. Limited capacity: each channel has limited capacity. Only a limited 
amount of processing can take place in the verbal or at the visual 
channel. 
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3. Active processing: Engaging in a task like learning requires sub- 
stantial cognitive processing on both channels. 
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Figure 3-5. Dual-coding theory (Paivio, 1986, 1991). 


More concrete evidence for the link between awareness and the work- 
ing memory comes from Endsley (1995): 


“Once perceived, [awareness] information is stored in working memory. In the absence 
of other mechanisms (such as relevant long-term memory stores, most of a person’s 
active processing of information must occur in working memory. New information 
must be combined with existing knowledge and a composite picture of the situation 
developed” (p.43). 


Further, Endsley (1995b) integrated her conception of situation aware- 
ness (cf. section 2.1.1) into the cognitive system. This formed the model 
of situation awareness in dynamic decision making (Endsley, 1995) (cf. 
Figure 3-6). It contains stages of perception, cognitive processing and ac- 
tions introduced in section 2.1.2. Self-awareness encompasses a subject’s 
mental model of the situation upon which all of his/her decisions rely 
(Endsley et al., 1998). 

As shown by Endsley (1995b) awareness involves a state of knowledge 
as well as dynamic processes of interaction with the environment. Aware- 
ness (the mental model) can be seen as a product of these processes and 
also as the “glue” connecting the three-staged process of perception, com- 
prehension and prediction. Yet, there are different opinions on whether 
information acquisition and maintenance processes belong to awareness 
itself or not (Endsley, 2015). 
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Figure 3-6. Model of situation awareness in dynamic decision making (Endsley, 1995). 


3.1.3 Forgetting and Cognitive Artifacts 


Forgetting is the inability to access information that was once stored and 
available to the subject (Ebbinghaus, 1885). Ebbinghaus sought to identify 
basic memory processes of the primary memory (i.e., the short-term 
memory, cf. section 3.1.2) that are independent of a subject’s past 
knowledge (i.e., the long-term memory). To do this, he used nonsense syl- 
lables as the items to be remembered and determined how many runs 
through a list of these it would take to recite a list perfectly. Based on the 
number of syllables and the number of views he constructed a diagram 
known as his learning curve (cf. Figure 3-7). The curve showed that the 
number of views increased rapidly when the number of required syllables 
exceeded 7 items. 
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Figure 3-7. Learning curve (Ebbinghaus, 1885). 


Another aspect he examined was forgetting for which he used another 
diagram - the forgetting curve (Ebbinghaus, 1885) (cf. Figure 3-8). The 
curve depicts the decline in memory retention over time when there is no 
attempt to retain it (e.g., by active recall/spaced repetition, better memory 
representation etc.). Ebbinghaus suggests repetitive rehearsals for better 
memory and especially emphasized the importance of sleep (Ebbinghaus, 
1885). This precursor to implicit memory testing was, by the way, repli- 
cated successfully by researchers in 2015 (Murre & Dros, 2015). 
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Figure 3-8. Forgetting curve (Ebbinghaus, 1885). 


There are basically two theories to explain Ebbinghaus’ observations 
and results, i.e., why people forget (Baddeley, 1999; Sternberg et al., 2011): 
1. Decay theory: information gradually fades over time (this is shown 

by a forgetting curve) 

2. Interference theory: information in memory is disrupted or ob- 
scured by subsequent activities following the initial perception. 
There are two kinds to be distinguished: 

a. Retroactive interference: newly acquired knowledge im- 
pedes the recall of older material. Here, the interference 
happens before the first recall. 

b. Proactive interference: material that was perceived or 
learned in the past impedes the perception/learning of 
new material. 


The measure of how easily a piece of information or chunk can be 
retrieved from memory is called activation (Budiu, 2014). It is influenced 
by three different factors: 

1. Practice/rehearsal: how many times a chunk has been used in the 

past 

2. Recency: How much time went by since the information has been 

used and ifit was the last or first in a larger amount of information 
(primacy/recency effect (Deese & Kaufmann, 1957)) 
3. Context: what is present in the subject’s focus of attention 


Cognition, Coordination & Coordination Support | 83 


These factors are also the reason why one cannot actively forget some- 

thing, no matter how hard they try as their trials are a kind of rehearsal 
that rather keeps things in memory. 
Besides the above-mentioned internal factors there are other forms of 
memory aids. Cognitive artifacts can be defined as “those artificial devices 
that maintain, display, or operate upon information in order to serve a 
representational function and that affect human cognitive performance” 
(Norman, 1991). They are things made by man to extend our cognitive 
abilities like check lists, calendars or a handkerchief acting as a reminder 
(cf. Figure 3-9). 


Figure 3-9. Cognitive artifacts (Source: Pixabay, CC0) 


But these tools also easily influence the way tasks get done. According 
to Norman (1991), they can... 


e distribute the task across time (precomputation). 
e distribute the task across people (distributed cognition). 
e change the actions required of the individuals doing the activity. 


That is, they not only support memory but due to their externalized 
form support distributed collaboration (cf. section 1.1). Therefore, they 
will evolve to coordinative artifacts later in section 3.2.2. For now, 
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cognitive artifacts “act as mediators between us and the world, both in 
execution (between actions and the resulting changes to the world state) 
and in perception (between changes in the world and our detection and 
interpretation of the state)” (Norman, 1991, p.18). In other words, they are 
central piece of Neisser’s perception-action cycle (cf. Figure 2-4) described 
earlier (cf. section 2.1.2). 


3.1.4 Problem Solving and Decision Making 


Problem solving and decision making are two closely linked cognitive 
tasks that both form the second quadrant (“Choose”) in McGrath’s 
(McGrath, 1984) Group Task Circumplex (cf. Figure 2-8 in section 2.2.2). 
Problem solving in general “involves mentally working to overcome ob- 
stacles that stand in the way of reaching a goal” (Sternberg et al., 2011, 
p.484). Basically, it is finding solutions to problems - also in collaborative 
settings. “Problem solving occurs when users do not know what to do 
next” (Ritter et al., 2014, p.175). Finding solutions turns out to be quite 
dependent on the specific context. According to Sternberg et al. (2011), 
there are two different types of problems (Sternberg et al., 2011, p.447): 
e  Ill-defined problems: no specific goals nor clearly expected solu- 
tions. 1? 
e Well-defined problems: allow for more initial planning, specific 
goals and clearly expected solutions plus a clear path to solution. 


People usually develop a problem solving strategies as part of a prob- 
lem solving cycle (Sternberg et al., 2011, p.444) (cf. Figure 3-10). In this 
cycle they will recognize the problem, define the problem, develop a strat- 
egy to fix the problem, organize the knowledge of the problem cycle, fig- 
ure out the resources at their disposal, monitor one's progress, and eval- 
uate the solution for accuracy in solving the problem. 


12 section 4.2 describes CSCW design later as wicked problem which is basically another term 
for ill-defined. 
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Figure 3-10. The Problem-Solving-Cycle (Sternberg et al., 2011). 


On the other end, “decision-making is the result of problem solving” 
(Ritter et al., 2014, p.183). More commonly, the term decision is used to 
denote an important problem of choice among multiple options where 
much is at stake (Laux et al., 2014). In decision theory the term decision 
encompasses all acts of choice. Here, a decision defines all (sub-)con- 
scious acts of choice among multiple alternatives for action no matter how 
big or how small. A decision-making problem can be characterized by the 
question “what option should be chosen from multiple alternative op- 
tions?” One option might also be not to change anything maintaining the 
status quo (Laux et al., 2014). In order to solve a decision problem or to 
make a choice every option has to be examined regarding its benefits and 
risks. Results which can be reached with an option have to be projected 
and assessed on their impact. Options, results and impacts mark the field 
of decision making. Further, the goals and objectives of the decision 
maker are required to validate the options (Laux et al., 2014). 
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Decision making can be conceived as a specific kind of problem-solv- 
ing process with the following steps: 
1. Problem definition 
Refinement of goals and objectives 
Examination of options 
Option selection 
Realization of decision 


N 


In many situations the time to make decisions is critical. The Hick- 
Hyman Law (sometimes only Hick’s law) (Hick, 1952; Hyman, 1953) de- 
scribes the time to make the decision in relation to the number of options 
available. But the number of choices does not merely increase response 
times but it does so in a logarithmic way. Hick’s law is already a concept 
that plays a role in HCI. It is usually used “to justify menu design deci- 
sions. For example, to find a given word (e.g., the name of a command) in 
a randomly ordered word list (e.g., a menu), scanning of each word in the 
list is required, consuming linear time, so Hick's law does not apply. How- 
ever, if the list is ordered alphabetically and the user knows the name of 
the command, he or she may be able to use a subdividing strategy that 
works in logarithmic time” (Landauer & Nachbar, 1985). Decision-making 
basically comes with a speed-accuracy trade-off. One can either be very 
careful and slow but precise in making decisions or very fast but less ac- 
curate. 

In this thesis, the decision-making is not to be seen as part of the pri- 
mary task, i.e., the group’s tasks to make a decision on a certain matter. 
Decision-making plays also small and mostly subconscious part in the 
secondary task of coordination! It is about deciding who is doing what 
next. Figure 3-6 shows how situation awareness facilitates decisions in or- 
der to decide what to do next. “Situation awareness forms the critical input 
to — but is separate from - decision making, which is the basis for all sub- 
sequent actions” (Endsley et al., 1998, p.1). 


3.1.5 Capacity, Cognitive (Work-)Load and Effort 


The MHP’s cognitive subsystem (cf. section 3.1.1) consists of store and 
processor that can be described in terms of capacity (number of units) and 


Cognition, Coordination & Coordination Support | 87 


cycle time (t). The storage capacity can be determined by the number of 
items (u) or as workload (relation of used capacity /available capacity). 

The most important characteristic for a processor is its cycle time (t) 
which denotes the amount of time needed to complete one task (e.g., re- 
trieving information from memory). Also, the processor can be assessed 
in terms of load, especially if it is able to work on more than just one task 
at a time. 

As mention earlier in section 3.1.2, “the primary characteristics of 
short-term memory are its low capacity and its volatility” (Johnson, 2010, 
p-82). Also, Ebbinghaus’ learning curve suggested a capacity limit. When 
it comes to memory and capacity almost every source of research refers to 
Miller (1956) and his chunks or rather the magical number seven plus 
minus two (Miller, 1956) whereas later research suggests rather four 
(Cowan, 2001). Miller concluded that the capacity of the short-term 
memory (STM) lies between five and nine meaningful items or chunks of 
information for a typical adult. In this case, the word “meaningful” refers 
to whether the person is able to find a way of relating the items to what he 
or she already knows. This process is called chunking (Sternberg et al., 
2011). It decides ifyou see 14 9 2 or 1492 or the year Columbus discovered 
America (Revlin, 2012). 

Turning to the cognitive processor, “one way of measuring [the central 
executive’s] capacity is through the working memory span task, which cor- 
relates with the comprehension capacity” (Baddeley, 2013, p.69). The 
working memory span is a task devised by Daneman and Carpenter (1980) 
(cf. Appendix C) which “involves presenting the subject with a series of 
sentences. The subject was required to read each one and then, after the 
final sentence, recall the last word of each sentence. Some people have 
difficulties remembering more than two, others easily remember four” 
(Baddeley, 2013, p.67). 

On the other end, “at the most general level, mental workload can be 
described as the relation between the function relating the mental re- 
sources demanded by a task and those resources available to be supplied 
by the human operator” (Parasuraman et al., 2008, p.145). The relation is 
expressed by two figures of capacity: the used capacity divided by the avail- 
able capacity (both in numbers of items) which yields a percentage. 
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Figure 3-11. Capacity and workload displayed in Windows task manager (Source: Christoph 
Oemig). 


This is also the third occasion where this thesis draws on a metaphor 
from computer science. This time the memory capacity is measured in 
Gigabyte (GB). For instance, the Windows task manager (cf. Figure 3-11) 
shows both, load 2.49/4GB and capacity of 4 GB. The workload of a CPU 
is calculated by the number of possible threads and how many of them 
currently work as opposed to the other fraction being idle. 

According to Mayer and Moreno (2003) there are three types of cogni- 
tive demands that influence the cognitive workload (p.45): 
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1. Essential processing: cognitive processes required to pick up, 
make sense and integrate presented information 

2. Incidental processing: cognitive processes that are not part of the 
essential ones, that deal with additional, unnecessary or redun- 
dant information (waste of resources) 

3. Representational holding: cognitive processes aimed at holding a 
mental representation in working memory over a period of time 
due to the suboptimal presentation of information 


Especially humans are subject to cognitive overload where the sub- 
ject’s intended cognitive processing exceeds the subject’s available cogni- 
tive capacity (Mayer & Moreno, 2003). “Reducing cognitive load can in- 
volve redistributing essential processing, reduce incidental processing, or 
reducing representational holding” (ibid, p.43). This corresponds to the 
Cognitive Load Theory (Chandler & Sweller, 1991) where essential pro- 
cessing is called germane load, incidental processing is extraneous load, 
and the representational holding corresponds to intrinsic load. Awareness 
and coordination belong to the group of essential processing. 

Last but not least, talking about effortless coordination involves the 
term effort which can be defined as “the conscious exertion of power or 
total work done to achieve a particular end” (Merriam-Webster, 2017). The 
typical unit for effort is time or more exactly time per one unit that is avail- 
able to work on something though in many cases the effort is expressed 
as total effort (O ’ Regan, 2017). It is also not to be confused with the du- 
ration of something. For instance, if a task takes a total effort of twelve 
hours and there is one unit available then the duration is twelve hours and 
the effort per unit is also twelve hours. If there are two units available the 
total effort is still the same but the duration is split by two as well as the 
effort per unit. 


3.2 Coordination Aspects 


In general, the term coordination is defined as “the harmonious function- 
ing of parts for effective results” or as “the process of organizing people 
or groups so that they work together properly and well” (Merriam-Webster, 
2016c) (cf. Figure 3-12). 
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Figure 3-12. Working together to achieve a common goal requires coordination (Source: Pix- 
abay, CCO). 


Coordination takes place when a group of people act or interact to- 
gether to achieve something (Preece et al., 2015, p.118), i.e., they either 
collaborate or cooperate as described in section 1.1 where cooperation re- 
quires more coordination and communication than collaboration. 

The symptoms of poor coordination include “people bumping into one 
another, duplicating actions that another person has just completed, or 
attempting to take shared resources at the same time” (Gutwin & 
Greenberg, 2000, p.118). 


3.2.1 Coordination Theory 


The definitions above serve as a first introduction. But as previous defini- 
tions of awareness, they are hardly the definition of a construct, i.e., a lot 
more details are required. However, for the case of coordination as op- 
posed to the one of awareness, there is an extensive concept known as 
coordination theory that formulates a construct of coordination. Its authors, 
Malone and Crowston (1990) (also later in Crowston et al. (2006)) describe 
this theory as 


“a body of principles about how activities can be coordinated, that is, about how actors 
can work together harmoniously. (...) In its attempt to find generalizations that apply 
across disciplines and across levels of analysis, coordination theory resembles earlier 
work on systems theory and cybernetics” (p.358). 
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Overall, coordination theory is the “body of principles about how the 
activities of separate actors can be coordinated” (Malone, 1988, p.6). As 
part of this theory they define coordination as “the act of managing inter- 
dependencies between activities performed to achieve a goal” (Malone & 
Crowston, 1990, p. 361). As shown for the case of awareness (cf. section 
2.2.2), coordination is another example of a secondary task: “Coordination 
is distinguished from production: (...) We divide goal-relevant tasks into 
two categories: coordination task and production tasks” (Malone, 1988, 
p-5). Coordination tasks are information processing tasks performed due 
to interdependencies. Production tasks are all the other tasks that are per- 
formed in order to achieve the goal. Interdependencies and goals are two 
of the following components of coordination: 

e Goals: coordination is always directed towards achieving a certain 
goal. 

e Activities: the goal is decomposed into activities which have to be 
coordinated. 

e Actors: activities are assigned to actors who execute them. 

e Interdependencies: those are the goal-relevant relationships be- 
tween activities. 


Interdependencies are the vital part of the theory that cause the need 
for coordination. 


“In our previous work, we defined coordination as something that occurs only when 
multiple actors are involved, Since then, however, we have become convinced that the 
essential elements of coordination listed above arise whenever multiple, interdependent 
activities are performed to achieve goals — even if only one actor performs all of them 
(...) If there is no interdependence, there is nothing to coordinate” (Malone & Crowston, 
1990, p.362). 


Further, the authors identify three types of generic interdependencies 
(cf. Table 3-1): 
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Table 3-1. Interdependency types. 


which is required 
by the next activity 


ered in time to be 
used 


Kind of interde- | Common object Example in manu- | Example of coordi- 
pendence facturing nation process 
Prerequisite Out of one activity | Parts must be deliv- | Ordering activities, 


moving infor- 
mation from one 
activity to the next 


Shared resource 


Resource required 
by multiple activi- 
ties 


Two parts installed 
with a common tool 


Allocating re- 
sources 


Simultaneity 


Time at which 
more than one ac- 
tivity must occur 


Installing two 
matched parts at the 
same time 


Synchronizing ac- 
tivities 


In another step Malone and Crowston (1990, 2006) specified the pro- 
cesses underlying coordination on different levels with coordination being 
the top-level process (pp.364) (cf. Table 3-2). 


Table 3-2. Processes underlying coordination (Crowston et al., 2006; Malone & Crowston, 
1990). 


Process Level Components Examples of Generic Processes 
Coordination Goals, activities, actors, | Identifying goals, ordering activities, as- 
resources, interdepend- | signing activities to actors, allocating re- 
encies sources, synchronizing activities 
Group decision- | Goals, actors, alterna- | Proposing alternatives, evaluating alter- 
making tives, evaluations, | natives, making choices (e.g., by author- 
choices ity, consensus, or voting) 
Communication Senders, receivers, mes- | Establishing common languages, se- 
sages, languages lecting receiver (routing), transporting 
message (delivering) 
Perception of | Actors, objects Seeing same physical objects, accessing 
common objects shared databases 


The above table shows that decision making is a vital aspect of coordi- 
nation. 

“Many coordination processes require making decisions that affect the activities of the 

group. For instance, in sharing resources a group must somehow ‘decide’ how to allo- 

cate the resources; in managing task/subtask dependencies, a group must ‘decide’ how 

to segment tasks” (Malone & Crowston, 1994, p.99). 


Cognition, Coordination & Coordination Support | 93 


The group decision-making processes in turn 


“require members of the group to communicate in some form about the goals to be 
achieved, the alternatives being considered, the evaluation of these alternatives, and the 
choices that are made. This communication requires that some form of ‘message’ be 
transported from senders to receivers in a language that is understood by both” (Malone 
& Crowston, 1990, p.364). 


Malone and Crowston (1990) describe communication as interacting 
with the environment which comes close to Neisser’s (1976) perception- 
action-cycle (cf. Figure 2-4 in section 2.1.2). 

Overall, this layered approach of coordination processes fits the one in 
the beginning of this thesis where collaboration builds on coordination 
which builds on communication (cf. Figure 1-6 in section 1.1). This will 
also play an important role later in chapter 1. 


3.2.2 Coordination Practice 


Now that is known what coordination is it is time to shift over to how 
coordination is actually done. “In general, collaborative activities require 
us to coordinate with each other (...) In particular, we need to figure out 
how to interact with one another to progress with our various activities. 
To help us we use a number of coordination mechanisms” (Preece et al., 
2015, p.118). 

Primarily, these include (Preece et al., 2015): 

e Verbal and non-verbal communication 

e Schedules, rules and conventions 

e Shared external (i.e., physical) representations 


The coordinative practices found by ethnographic studies, named 
“monitor and display” (cf. section 2.2.1) are a prime example for verbal 
coordination mechanisms. An example for non-verbal communication is 
a conductor of an orchestra or marching band (cf. Figure 3-13). 
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The conductor’s baton and his movements are the means to display 
the necessary information to all the musicians involved. At the same time 
this is an example of the interdependency type simultaneity as the con- 
ductor synchronizes the activities of each individual musician. 


Figure 3-13. A conductor coordinates the sounds of the orchestra using hand signals for non- 


verbal communication (Source: Pixabay, CC0). 


As another example, our daily traffic, i.e., a large number of drivers 
and their respective vehicles, is coordinated by rules and conventions. 
These rules and conventions are implemented as traffic signs and traffic 
lights along the streets (cf. Figure 3-14). As traffic lights synchronize the 
ongoing driving at an intersection, they are another example of the inter- 
dependency type simultaneity. 
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Figure 3-14. Traffic lights as coordination mechanism implementing rules for traffic 
(Source: Pixabay, CCO). 


In general, the interdependency type simultaneity deals with resources 
that are potentially used by multiple users at the same time. On the other 
end, the interdependency type shared resource deals with resources that 
can only be used by one user at a time requiring a coordination mecha- 
nism that enforces asynchronous utilization. For instance, the emergency 
intercom system in German regional trains allows only one participant to 
talk at a time - a transmission mode referred to as half-duplex (Couch, 
2013). Whether a participant may talk is signaled by a traffic light meta- 
phor using red and green lights in addition to a white push-to-talk (PTT) 
button (cf. Figure 3-15). 
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Figure 3-15. Intercom system in German regional trains using a half-duplex transmission 
mode. The user is required to wait with the red light and may talk when the green light 
flashes (Source: Christoph Oemig). 
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Another example of a coordination mechanism are schedules which 


can be found, for instance, at airports, train stations or at schools telling 
people where and when to go (cf. Figure 3-16). 
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Figure 3-16. A schedule as coordination mechanism (Source: Pixabay, CC0). 
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Recipes in a cook book (cf. Figure 3-17) are a different type of plan. 
These define the ingredients and the sequence of steps, i.e., the ingredi- 
ents are not all added at the same time (indicating simultaneity) but in 
distinct steps one after the other following a pre-defined order. Thus, one 
step is the prerequisite for the next making it an example for a verbal co- 
ordination mechanism for the interdependency type prerequisite. 
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Figure 3-17. Recipe using verbal communication as coordination mechanism for the inter- 


dependency type prerequisite (Source: Pixabay, CCO). 


Other examples of coordination mechanisms use physical representa- 
tions (later referred to as coordinative artifacts). For instance, the key 
board at a hotel’s reception helps the manager to coordinate the hotel’s 
rooms as shared resources (cf. Figure 3-18). The key chain is a physical 
representation for one particular room identified by a room number. 
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Figure 3-18. Hotel keys as physical representation for the rooms (Source: Pixabay, CC0). 


All in all, this shows that a coordination support system (e.g., the traffic 
light) may implement multiple coordination mechanisms to address one 
interdependency type (cf. Table 3-3). 
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Table 3-3. Coordination mechanism examples per interdependency type. 


Interdependency type 
Coordination mech- | Prerequisite Shared Resource Simultaneity 
anism 
Verbal/non-verbal Recipe, Reservation list Conductor, 
communication Process descrip- Traffic lights 
tion 
Schedules, rules, | Recipe, Flight schedule Conductor, 
and conventions Process descrip- Traffic lights 
tion 
Shared external rep- | Recipe, Hotel keys Conductor, 
resentation Process descrip- Traffic ligh 
tion 


The above describes a number of real-world coordination support sys- 
tems. In the area of computer-supported cooperative work computational 
coordination support is used. 


“In cooperative work settings characterized by complex task interdependencies, the ar- 
ticulation of the distributed activities requires specialized artifacts which, in the context 
of a set of conventions and procedures, are instrumental in reducing the complexity of 
articulation work and in alleviating the need for ad hoc deliberation and negotiation” 
(Schmidt & Simone, 1996, p.162). 


A computational coordination support system (as opposed to the ones 
shown above) should be conceived as a specialized piece of software inter- 
acting with the user articulating work-related information (i.e., awareness 
information via awareness cues, cf. section 2.3.1) using specific coordina- 
tion mechanisms (Preece et al., 2019): 


“a coordination mechanism is a construct consisting of a coordinative protocol (an inte- 
grated set of procedures and conventions stipulating the articulation of interdependent 
distributed activities) on the one hand and on the other hand an artifact (a permanent 
symbolic construct) in which the protocol is objectified” (Schmidt & Simone, 1996, 
p.165). 


In more detail, a coordinative protocol is the part of a coordination 
mechanism that supports situated action by reducing “complexity of ar- 
ticulating cooperative work by providing a precomputation of task inter- 
dependencies which actors, for all practical purposes, can rely on to reduce 
the space of possibilities by identifying a valid and yet limited set of 
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options for coordinative action in any given situation” (Schmidt & 
Simone, 1996, p.174). On the other hand, the other part of a coordination 
mechanism is the artifact that is used to “to objectify and give permanence 
to the coordinative protocol so that its stipulations are unceasingly pub- 
licly accessible” (Schmidt & Simone, 1996, p.176). This means it is a 
proxy, representing the state of the protocol. 

As indicated above, this basically makes awareness cues coordination 
mechanisms. In other words, coordination mechanisms in the digital 
world are nothing but a part of a coordination support systems for which 
Schmidt and Simone (1996) defined the subsequent requirement: they 
have to be flexible. “It must be constructed in such a way that actors are 
supported in controlling the propagation of changes to the protocol with 
the cooperative work arrangement” (ibid., p.187). 

The role of coordinative artifacts appears to have become topical in 
more recent years. For instance, in architectural practice (Schmidt & 
Wagner, 2002) they are part of the material work settings. They populate 
them playing “a crucial role in the seamless and effective coordination and 
alignment of cooperative work” (ibid., p.257). They are similar to but at 
the same time exceed the aforementioned cognitive artifacts (cf. section 
3.1.3) by not just being “vehicles for information” (ibid., p.257) or for 
memory aids. 

Redaelli and Carassa (2015) studied coordinative artifacts at an Italian 
airport. In doing so, they tried to understand “the role of plans in support- 
ing coordination at work by proposing an approach that focuses on how 
the plan’s capacity to anticipate interdependencies at work is maintained 
in face of changes in surroundings to ensure stability in the coordination 
of different stakeholders” (ibid., p.165). They found that artifacts antici- 
pate future ways of performing activities. Anticipation itself plays a key 
role in coordination. 

Bardram and Bossen (2005) studied how people in a hospital coordi- 
nate their work using a wide range of non-digital artifacts like white- 
boards, schedules, examination sheets etc. “The role of these artifacts in 
collaborative work is to lessen the amount of ‘articulation work’” (ibid., 
p.169). Articulation work (Strauss, 1985) is the effort of coordination tasks 
and responsibilities between distributed collaborators. This type of work 
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can be reduced by developing divisions of work, conventions and by the 
use of coordinative artifacts. 

Recently, Zaitsev et al. (2020) conducted a case study on coordination 
artifacts in agile software development. Here, coordination artifacts were 
used to mitigate coordination challenges. They identified four different 
types of coordination artifacts that vary in terms of their information rich- 
ness and mutability: 

1. Foundational 

2. Projective 

3. Exposition 

4. Indicative 


3.2.3 Systems for Coordination Support 


Already the introduction of this thesis defined coordination support as a 
standard support function for computer-supported cooperative work. The 
3C Model (Teufel et al., 1995) defines the group of systems closest to this 
area as workflow management systems (cf. Figure 1-8 in section 1.1). These 
are systems “where work activities are given a formal representation in 
terms of some workflow model which often stipulates how the contribu- 
tions of different participants are to be coordinated” (Sandor et al., 1997). 
Malone (1988) and later Grudin and Poltrock (2020) refer to these systems 
that help people to coordinate their activities as “coordination technology”. 
“Coordination technologies employed in the workplace such as meeting support sys- 
tems, group calendars, workflow management systems, and computer-aided software 
engineering systems were an early focus of CSCW. They gave way to studies of how 
people coordinate in the absence of (or despite) coordination management technologies. 
For example, Bowers et al (1995) studied the problems that deployment of workflow 
technology created in a large printing enterprise. Social networking also enables a new 


generation of coordination technologies whether mobile and location-aware real-time” 
(Grudin & Poltrock, 2020). 


Abbott & Sarin (1994) describe two different types of workflow sys- 
tems: 
1. Document-oriented workflow: The definition of routes is attached 
to shared documents. 
2. Activity-oriented workflow: The processes are modeled as se- 
quence of activities and documents are attached to activities. 
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Further, Gross & Koch (2007) distinguish two kinds of coordination 
support (p.91): 


1. 


Explicit coordination support (automation): The system is in 
charge of coordinating resources, i.e., the system decides, the user 
follows (cf. the traffic light in Figure 3-14). 

Implicit coordination support: The system offers (awareness) in- 
formation and the users make the decisions following a social or 
technical protocol. 


The degree of automation is basically a design choice which has to be 
carefully made. 


“Other systems acknowledge the need for basic coordination mechanisms by providing 


locking and versioning of shared documents based on check-in- and check-out opera- 


tions performed by users. These systems do not dictate behaviour, but simply coordinate 


concurrent access and provide status information about other users’ activities” (Glance 
et al., 1996, p.180). 


Based on Malone’s coordination theory (cf. section 3.2.1) three types 
of coordination support systems can be derived (Gross & Koch, 2007, 


p.91): 
1. 


Task sequencing systems (workflow management systems) ad- 
dressing the interdependency type of prerequisite. 

Access to shared resources: optimistic/pessimistic locking as well 
as concurrency control. When missing it could lead to thread in- 
terference and memory consistency errors. Badly implemented 
synchronization leads to thread contention, starvation, livelocks, 
and deadlocks (Dix et al., 2004; Lea, 1999). All of the above refers 
to the interdependency type of shared resource. 

Activity synchronization is needed in the case of the interdepend- 
ency type simultaneity. Systems in this area introduced a range of 
floor control mechanisms such as chair people, reservations and 
token-passing (Cook & Lunt, 1992; Crowley et al., 1990; Sarin & 
Greif, 1988). Also the spatial model introduced in section 2.3.2 
sought to minimize hard-wired constraints of traditional floor 
control (Benford & Fahlen, 1993, p.111). 
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Last but not least coordination support systems can be split in two 
groups depending on whether they can be used synchronously or 
asynchronously (as a reduced form of Johansen’s (1988) Time-Space 
Taxonomy, cf. Figure 1-7 in section 1.1). As with awareness support 
systems (cf. section 2.3.2) there appears to be an application level and a 
coordination level, meaning that there are basic coordination 
mechanisms like floor control, that can be implemented and provided in 
different ways in different applications, yet always with the purpose of 
activity synchronization. Again, this makes these basic mechanisms 
independent of the application context! 

When it comes to actual systems that support coordination it becomes 
obvious that most of them exist already outside of CSCW research (e.g., 
workflow management systems, calendar applications etc.). For instance, 
the three tools from the introductory section of this thesis use multiple 
mechanisms to address the different interdependency types of coordina- 
tion. The following systems serve the purpose of pointing out further as- 
pects when dealing with coordination support. They do not aim to provide 
a complete list of tools. 

When it comes to coordination support, a by this time quite old but 
often cited example is the COORDINATOR (Flores et al., 1988). It is a speech- 
based system (not to be confused with systems like Amazon Alexa!) that 
helps people make and keep track of requests and commitments to each 
other. It thus supports “mutual agreeing”, an important part of the task 
assignment process. Therefore, it uses language related to the underlying 
theory of speech acts (Searle, 1969). Communication is the primary unit 
to interaction (cf. section 1.1). “In COORDINATOR, users add structure to 
their messages based on conversational moves and actions, such as ‘re- 
quest’ and ‘notification’” (Henninger, 1991, p.25). The intention is to lead 
the user to think about what s/he is doing and then to characterize a par- 
ticular communication as one of several choices (e.g., make a request, 
make a promise etc.). However, the system failed: It imposed to many 
restrictions and users did not use the system as intended. In most cases 
they only used the first menu option “request” (Bullen & Bennett, 1990). 


13 https://alexa.amazon.com 
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Another system well known is this area was Lotus NOTES1* which was 
available as a commercial system. It was a document-based system for 
which flexible workflows could be implemented. Both were bundled as 
applications inside Lorus NOTES providing developers a high degree of 
flexibility. One type of these documents were emails for which NOTES pro- 
vided a tight integration. While being one of the most flexible systems, 
Lotus NOTES failed as well: 


“the findings suggest that where people’ mental models do not understand or appreciate 
the collaborative nature of groupware, such technologies will be interpreted and used as 
if they were more familiar technologies, such as personal, stand-alone software (e.g., a 
spreadsheet or word processing program). The findings further suggest that where the 
premises underlying the groupware technology (shared effort, cooperation, collabora- 
tion) are counter-cultural to an organization’s structural properties (competitive and in- 
dividualistic culture, rigid hierarchy etc.) the technology will be unlikely to facilitate col- 
lective use and value” (Orlikowski, 1992, p.362). 


Halloran et al. (2002) found similar issues when their students simply 
rejected the tool in favor of others, they were familiar with. The tool’s high 
degree of flexibility (as opposed to the COORDINATOR’s restrictiveness) was 
the reason for its failure. In many companies LOTUS NOTES was reduced 
to a simple email tool because that was what people were familiar with. 

Another more recent commercial tool is JIRA, probably the #1 software 
development environment used by agile teams by Atlassian Inc.!? (cf. Fig- 
ure 3-19). 

JIRA’S core concept is that of an issue tracker. The key component is 
an issue that might be a task, user story, defect or service request. Each 
issue has its own workflow attached to it, that is, it uses an activity-oriented 
workflow model (Abbott & Sarin, 1994). 


14 Lotus Notes was procured by IBM in 2013, then renamed to IBM Notes. In 2019 IBM sold 
Notes to HCL. 


15 https://www.atlassian.com/software/jira 
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Figure 3-19. JIRA von Atlassian Inc. (Source: Atlassian Inc.). 


The remainder of the tool consists of powerful filters based on a dis- 
tinct but intuitive query language and sets of issues like boards, sprints, 
reports and releases. The basic mechanisms are provided, yet, these follow 
the users’ concepts and language. Additionally, they are flexible enough 
to be used in many situations. 

Last but not least comes an example of a very lightweight coordination 
support systems that helps during synchronous conversations (thus sup- 
porting the interdependency type simultaneity): the typing indicator (cf. 
Figure 3-20). It helps to coordinate the turn-taking in a conversation using 
an implicit coordination support approach (Gross & Koch, 2007). Le., 
though the indicator shows, that the other is typing one can start typing 
as well. If it was an explicit coordination support approach the application 
would probably block all others, when somebody starts typing. 
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Figure 3-20. Coordination of synchronous communication (Source: Christoph Oemig). 


3.3 Challenges, Reflections and Criticisms 


As in the previous chapter, there are some issues in this chapter worth 
discussing or for taking a second look. They are all organized in short sec- 
tions for easier reference later. 


3.3.1 Forgetting versus Overfitting 


In most cases forgetting has not a positive reputation as it is connected to 
information, memory or data loss. Memories are lost typically due to ill- 
ness, accident or simply age (Sternberg et al., 2011). However, forgetting 
is a vital function to the mind’s health and well-being as it basically solves 
its capacity problems. Information not needed becomes inaccessible to 
make room for newer memories (cf. section 3.1.3). Overall forgetting is 
an important memory function that keeps humans from overloading 
(Johnson, 2010). 
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At the same time, it also keeps the memory from overfitting. This is 
another finding from computer science explaining cognitive issues, this 
time from the field of artificial intelligence (AI). It describes the state 
“when a mathematical model is so good at matching the data it has been 
programmed with that it is unable to predict which data might come next” 
(Gravitz, 2019). Recent research even suggests Alzheimer being a prob- 
lem of forgetting and not remembering (Gravitz, 2019) also relating it to 
overfitting. 

The prediction of future states is a core element of situation awareness 
(cf. section 2.1.2) and of coordination (cf. section 3.2.1). Thus, overfitting 
imposes also a threat to these capabilities. On the other hand, awareness 
and coordination support becomes the key technology to prevent it. 


“Unlike previous communication technologies, however, the new computer-based tech- 
nologies also have the potential to transfer information more selectively. Thus (...) new 
coordination technologies have the potential to help reduce information overload by di- 
recting information more accurately to people who want to know it without overloading 
others” (Malone, 1988, p.13). 


The design goal of awareness and coordination support must not be to 
help people to remember everything, but to remember the right things at 
the right time. This becomes even more important when dealing with dy- 
namic awareness and coordination support (cf. section 2.3). Here, it raises 
the following questions: What information is needed when? What is the 
best type of transition between contexts? What is the effort of such transi- 
tion? Again, these are all use-inspired basic research questions. 


3.3.2 Shared Capacity 


Section 3.1.5 talks about the memory’s capacity and workload from a ge- 
neric perspective. Section 2.2.2 in the previous chapter introduced pri- 
mary and secondary tasks for the context of cooperative work. Combining 
the two in terms of capacity, reveals a major difference between single 
user tasks and group tasks (cf. Figure 3-21). 


Cognition, Coordination & Coordination Support | 109 


Single user task Group task 


Secondary task knowledge 
(coordination) 
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E Secondary task knowledge 
Wo rking (awareness) 
Me mory Primary task knowledge 
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Primary task knowledge 


Figure 3-21. Working memory capacity for single user task and group task. 


While in single user tasks the primary task knowledge may occupy 
nearly the entire capacity alone, it has to share this capacity with the 
knowledge for the secondary tasks of awareness and coordination during 
a group task no matter if the scenario is a computer-supported setting or 
not. This raises questions like what is the memory ratio in terms of pri- 
mary to secondary task knowledge? Another aspect is (again) forgetting — 
as just mentioned in the previous section. How does forgetting affect pri- 
mary and secondary task knowledge? Do the same mechanisms apply for 
all knowledge types (primary and secondary) the same way or are they 
treated differently? Again, these are further use-inspired basic research 
extending upon section 2.4.1. 


3.3.3 Working Memory and Long-Term Memory 


Earlier in this thesis, awareness was said to be a mental model (cf. section 
2.1.2). However, while awareness is said to be stored in the working 
memory (cf. section 3.1.2) the previous statement appears odd as mental 
models originally belonged to the long-term memory construct (Craik, 
1943). In hindsight, these statements appear hard to be matched. Endsley 
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(2000) admits: “to view SA as a function of either working memory or 
long-term memory would probably be erroneous” (Endsley, 2000, p.14). 
However, there actually is research, e.g., (Johnson-Laird, 1983), that pro- 
claims mental models being part of the working memory. Other, more 
recent research (Jones et al., 2011) acknowledges both positions and ac- 
cepts them as equally true. Endsley explains her insight (and deviation 
from her earlier statement (cf. section 3.1.2)) with the decay of memory 
that did not strike in tests although expected and with an alternate model 
of cognition that conceives the working memory to be an activated subset 
of the long-term memory (Cowan, 1988) (cf. Figure 3-22). 


Attention 


— Long-term Memory 


Sensory Input 


Figure 3-22. Mental model: working memory versus long-term memory (Cowan, 1988; 
Endsley, 2000). 


“An increasing number of alternative accounts has emerged subsequently (...) One of 
the principal questions concerns the relationship between working memory and LTM. 
Baddeley and Hitch (1974) assumed that the two were separate systems. However, a 
number of authors take a different view, maintaining that working memory corresponds 
to an activated region of LTM” (Braisby & Gellatly, 2005, p.316). 
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As oftoday, this does not contradict the current working memory con- 
struct by Baddeley (2000, 2012) anymore as the component ofthe episodic 
buffer was introduced in 2000 which has a link to the long-term memory 
(LTM) (cf. Figure 3-4). In terms of the assessment approach described 
later in this thesis the exact construct at this level makes no difference. 


3.3.4 Cognitive versus Social Science 


All the cognitive concepts and ideas of this chapter make sense for the 
context of awareness and coordination. How come that there appears to 
be a problem with cognitive science in CSCW? Especially in its early days, 
cognitive science earned itself a reputation of only dealing with single us- 
ers in laboratories, i.e., artificial settings. “Much evaluation ofthe past was 
concerned with the cognitive functioning of a single user sitting alone in 
front of a computer display” (Neale et al., 2004, p.112). This especially was 
criticized in the context of CSCW where early systems failed and social 
aspects became increasingly dominant, for instance, through ethno- 
graphic studies (cf. section 2.2.1). “Indeed, in the last few years the carica- 
ture of HCI notions of isolated ‘users’ and tasks’ has become a popular 
form of critique both in the social sciences and among disillusioned per- 
sons within the HCI community itself” (Rogers & Ellis, 1994, p.121). Ex- 
tending on section 2.4.3, this might explain why there is hardly any cog- 
nitive research as part of CSCW. 


“With the emergence of the new field of CSCW in the mid-to-late [19]80’s, the problem 
of how computer systems should be designed to support groups of people communi- 
cating and working together came to the forefront of research on system design. Cogni- 
tive psychologists, however were left behind. Simply, the cognitive models and task an- 
alytic tools developed in HCI were unable to be applied in this context because of the 
constraints that follow from their inherent individual-based focus. Sociologists and an- 
thropologists attempted to fill this lacuna and thus began to make their mark in this area 
of research. Inspired by the influence of Suchman’s work in the HCI community, a 
number of social scientists started to conduct ethnographic studies of technology-medi- 
ated collaborative work” (Rogers & Ellis, 1994, p.122). 


Social and organizational concerns then dominated the research 
agenda for studying collaborative work and CSCW. With the cognitive 
researchers protesting: “On the one hand they assume a decontextualised 


112 | Cognition, Coordination & Coordination Support 


model of the actors while on the other they neglect the cognitive 
capabilities of these actors” (Rogers & Ellis, 1994, p.119). 


“the omission of cognitive aspects in these analyses is, we propose, a grave mistake. 
Given that much work activity is inherently cognitive (e.g., people need to think, solve 
problems, predict, make decisions and so on) it is argued that there needs to be an un- 
derstanding of how work activities are performed at this level of analysis in order to 
design computer systems that can support both cognitive activities and social interac- 
tions. The problem, however, is how one might start to analyse the cognitive activities of 
people in their working environment rather than the workings of the isolated mind of a 
single individual. To this end, the alternative approach of distributed cognition is pro- 
posed” (ibid, p.123). 


In the following, cognitive scientists suggested a new approach analyz- 
ing and explaining collaborative work known as distributed cognition 
(Hutchins, 1995; Hollan et al., 2000): 


“[It] attempts to overcome the limitations of existing single-disciplined frameworks, for 
studying collaborative work, by traversing conventional disciplinary boundaries. In par- 
ticular, it is intended to explain socially distributed, cognitive work activities that are 
mediated by the rich assortment of technological artefacts found in the workplace” 
(Rogers & Ellis, 1994, p.125). 


However, this approach got rejected as well: 


“Following recent developments in the psychology of work, we might conceive of this 
organisation as a form of ‘distributed cognition’; a process in which various individuals 
develop an interrelated orientation towards a collection of tasks and activities (cf. 
Hutchins 1989, Olson 1990, Olson and Olson 1991). And yet, even this relatively radical 
reconceptualisation of the relationship between the individual, his or her activity and the 
system, does not quite capture the situated and socially organised character of coopera- 
tive work (...) Whether one subscribes to a theory of distributed cognition or a more 
sociological conception of cooperative work, it is clear that we need to move away from 
laboratory studies of cognition, ‘which have deliberately stripped away the supporting 
context of the everyday world, in an effort to study ‘pure internal processes’ (Olson 1990), 
and begin to explore task coordination and computer support in real world everyday 
work settings” (Heath & Luff, 1992, p.71). 


The argument of not understanding social details followed the allega- 
tion of irrelevant and commonsensical research results lacking practical 
implications on the other side: 
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“Many ofthe approaches advocated by social scientists [cf. section 2.2.1] for informing 
CSCW design similar to those that have been applied by the cognitive sciences to inter- 
face design for single user systems in HCI, such as design recommendations and build- 
ing software prototypes and evaluation tools. So far there have been very few attempts 
to translate findings from workplace studies beyond the provision of a few general de- 
sign recommendations. Moreover, fieldworkers were only too aware that their practical 
offerings are meagre and commonsensical compared with their rich and poetic accounts 
of the workplace. Whereas HCI researchers have found numerous ways of transforming 
their findings into practical implications and formal prescriptions that, arguably, have 
proved useful for designers, CSCW researchers are finding it more difficult to follow 
suit” (Plowman et al., 1995, p.320). 


Following this dogmatic battle, it is hard to find further cognitive work 
in CSCW. In 2002, the social research agenda found itself stuck with the 
“Problem of Awareness” (Schmidt, 2002) (cf. section 2.4.5). In 2004, Neale 
et al. found the “the evaluation of distributed CSCW systems has been 
too frequently method driven by various disciplinary preferences, rather 
than driven by frameworks that get the appropriate questions answered” 
(p.112). “Reconsidering Awareness” in 2016 Schmidt and Randall (2016) 
diagnose a lack of consensus regarding awareness and coordination re- 
sulting in a research field exhibiting a fragmented state (Schmidt, 2009) 
others referred to as “The Big Hole” (Kostakos, 2015). 

On the other end, cognitive science today employs “theoretically based 
constructs that are evaluated in their generality through empirical studies 
in a wide variety of laboratory tasks, simulators, ‘microworlds’, and actual 
work domains” (Parasuraman et al., 2008, p.140) - yet all outside of 
CSCW. 


3.3.5 The Problem with Explicit Coordination Support 


Supporting coordination has its limits. Especially when the system is to 
take over (e.g., as with traffic lights) as done with explicit coordination 
support (cf. section 3.2.3), designers have to be careful: “When designing 
coordination mechanisms, it is important to consider how socially ac- 
ceptable they are to people. Failure to do so can result in the users not 
using the system in the way intended or simply abandoning it” (Preece et 
al., 2015, p.122). The rigid mechanisms of coordination support were one 
of the reasons why CSCW applications failed (Grudin, 1988). “A key part 
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is getting the right balance between human coordination and system co- 
ordination. Too much system control and the users will rebel. Too little 
control and the system breaks down” (Preece et al., 2015, p.122). This re- 
veals a classic design tension. For instance, coordination facilities with the 
COORDINATOR were experienced as excessively rigid (Winograd & Flores, 
1986). But the COORDINATOR is not the only system where this occurred. 
Glance et al. (1996) strive for an explanation: 


“The promise of workflow solutions for coordinating organizational processes is cur- 
rently being obscured by strong criticism of the rigidity of their work representation. 
This rigidity arises in part from viewing work processes as unfolding along a single line 
of temporally chained activities. In reality, work evolves both horizontally, in the coop- 
eration of causally unrelated, but information-sharing tasks, and vertically, in the coor- 
dination of causally-dependent activities” (Glance et al., 1996, p.180). 


However, as will be shown later, there are further reasons why explicit 
coordination support should be avoided especially in cooperative contexts. 


3.3.6 Awareness as Support Function to Coordination 


This thesis started out with collaborative settings, then added coordination 
and coordination support to the picture (cf. section 1.1). When researchers 
in CSCW took over, they introduced the concept of awareness to facilitate 
coordination without going into further details about their relationship. 
“Awareness aids both fine and coarse-grained coordination, since it in- 
forms participants about the temporal and spatial boundaries of others’ 
actions, and since it helps them fit the next action into a stream” (Gutwin, 
1997, p.54). However, this chapter showed how awareness is the basis for 
coordination because it feeds the problem-solving and decision-making 
processes of coordination with the information needed. For the case of 
implicit coordination support, the coordination is then done by the subject 
which provides a high degree of flexibility. In case of explicit coordination 
support, the awareness information would be used by the system to coor- 
dinate the activities. 

Another aspect was the degree of coupling which was also used to ex- 
plain the difference between collaboration and cooperation or the differ- 
ence between workgroup and workflow systems (Neale et al., 2004; 
Piepenburg, 1991). 
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“In contrast [to workflow systems], in many awareness-oriented systems, the coordina- 
tion between different activities is supported by giving participants an awareness of what 
each other are doing or have done so that participants can coordinate their work them- 
selves. Many researchers would hope that, not only does this provide a ‘truer’ and more 
‘lightweight’ sense for ‘support’, but would also make for more flexible applications 
which are not liable to the usability criticism that can be made of more procedural-ori- 
ented approaches to CSCW” (Sandor et al., 1997, pp.221). 


Taking a closer look reveals that the above-mentioned flexibility has 
also something to do with the distribution of tasks among system and user 
as the user is always the more flexible. That means, if the highest degree 
of flexibility is needed then the problem-solving and decision making of 
coordination is left to the user. Handing problem-solving and decision 
making over to the system takes away flexibility (cf. section 3.2.3) - but 
also effort! Finding the right degree of how to distribute coordination work 
among user and system is another design tension indicating (again) the 
need for more use-inspired basic research (cf. section 2.4.2). This is espe- 
cially worth considering when trying to reach effortless coordination in 
cooperative systems because the lowest common denominator when 
building coordination support is to support awareness. The AETHER 
model did exactly offer this (cf. Figure 2-16 in section 2.3.2) by allowing 
direct and indirect access to its awareness engine. 

In summary, coordination is the higher goal and awareness supports 
especially the user to reach it. From that perspective awareness is a sup- 
port function to coordination (either done by the users themselves or by 
the system) which is a support function to cooperation. 


3.3.7 Mapping Awareness to Coordination 


The insights from the previous section, i.e., the general connection be- 
tween awareness and coordination, allows further conclusions: If aware- 
ness is a support function to coordination then there is... 


1. ... no awareness without the purpose of coordination. 

2. .. an explanation or mapping for each type of awareness (cf. sec- 
tion 2.1.1) using the coordination interdependency types (cf. sec- 
tion 3.2.1) 


The mapping of awareness types to coordination interdependency 
types is shown in Table 3-4. 
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Table 3-4: Mapping awareness to coordination interdependency types. 


Awareness Type 


Coordination Interdependency Type 


Availability awareness 
Background awareness 
Collection awareness 
Environmental awareness 
General awareness 
Group awareness 
Group-structural awareness 
Informal awareness 
Organizational awareness 
Peripheral awareness 
Presence awareness 
Situation awareness 
Task-oriented awareness 
Workspace awareness 


Shared resource 


Activity awareness 
Conversational awareness 
Document awareness 
Group awareness 

Mode awareness 

People awareness 
Perspective awareness 
Presence awareness 
Self-awareness 

Situation awareness 
Spatial awareness 
Task-oriented awareness 
Workspace awareness 


Simultaneity 


Background awareness 
Collection awareness 
Environmental awareness 
General awareness 
Group awareness 
Group-structural awareness 
Informal awareness 
Organizational awareness 
Peripheral awareness 
Presence awareness 
Process awareness 
Situation awareness 
Task-oriented awareness 
Workspace awareness 


Prerequisite 
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Only the awareness types of (un-)intentional awareness, mutual aware- 
ness and passive awareness could not be placed in the table as they are 
rather related to attention than awareness (cf. section 3.1.2). 

Another issue that becomes obvious is the relation of the coordination 
interdependency types to the Time-Space Taxonomy (cf. Figure 1-7 in sec- 
tion 1.1) and synchronous and asynchronous awareness information (cf. 
Table 2-4 in section 2.3.1). The types of prerequisite and shared resource 
are relevant in asynchronous modes independently of the location. On the 
other hand, simultaneity is relevant in synchronous modes, yet also inde- 
pendent of the place of interaction (cf. Figure 3-23). 


Same Time Different Times 


Prerequisite / 
shared resource 


Same Place Simultaneity 


Prerequisite / 
shared resource 


Different Places Simultaneity 


Figure 3-23. Interdependency types related to time and space. 


All in all, it was shown that awareness support (cf. section 2.4.7) and 
coordination support (cf. section 3.2.3) are independent of application 
context. What both depend on is the interdependency type! 


3.4 Summary 


This chapter picked up the two most important sidetracks that belong to 
the foundation of this thesis: cognition and coordination. Awareness and 
coordination have constantly been described as mental models and sur- 
rounding processes. This chapter uncovered what that actually means. 
While CSCW remains on the level that “awareness facilitates coordina- 
tion” this chapter elaborated their more detailed connection and 
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relationship. This chapter provided the main clues to a measurement of 
efforts in this context which is required when trying to reach the goal of 
effortless coordination. All of this underlines the importance of cognitive 
science to CSCW. 

On the other end the term coordination is backed up by a solid theory 
which was used to identify the main attributes and process. The resulting 
construct of coordination can be found in Appendix D2. 

This chapter’s main findings are: 


The limited capacity of the working memory is shared among pri- 
mary and secondary tasks (awareness and coordination). 
Awareness and coordination are two distinct secondary tasks that 
share common cognitive features like the working memory thus 
splitting further the available capacity. 

There are links to both working memory and long-term memory 
that provides information or input to problem-solving and deci- 
sion making. 

Though the conception of cognitive science has been attacked in 
the past especially concerning its contribution to the field of 
CSCW, this chapter shows that cognitive science offers a solid con- 
tribution which helps to escape the problems caused by solely fo- 
cusing on social issues. 

Awareness is a support function to coordination which is a sup- 
port function to collaboration and cooperation. 

Especially coordination support provides a new design challenge 
as it can be implemented on the system or user side, i.e., resulting 
in the system or the human making the decisions. Early systems 
already revealed that systems taking over coordination (explicit co- 
ordination support) is not well liked by users. 

All coordination activity and thus also all awareness efforts are di- 
rected towards one out of three interdependency types: simultane- 
ity, shared resource or prerequisite. 

This extends the notion of the previous chapter were the aware- 
ness support system is independent of the context of use to that 
the awareness support system is dependent on the coordination 
interdependency type. 
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Last but not least: as much as the term awareness was a hyped topic in 
CSCW and at the same time a highly elastic term causing a lot of recon- 
ceptualization up to highly philosophical discussions, the opposite is true 
for the term coordination which appears to be a lot more solid and sound. 

The missing link that is left towards a measurement approach for co- 
ordination and awareness is provided by the next chapter that takes a 
deeper look at the design and evaluation of their respective support sys- 
tems. 


4 Design & Evaluation of Cooperative Applications 


As stated at the beginning of this thesis, the design of groupware lies at 
the core of CSCW (cf. section 1.1). In general, “CSCW should be conceived 
as an endeavour to understand the nature and requirements of coopera- 
tive work with the objective of designing computer-based technologies for 
cooperative work arrangements” (Schmidt & Bannon, 1992, 2013, p.11, 
p.351). 

The design and evaluation are typical steps in a software development 
life cycle (SDLC). This development life cycle may differ in the process 
model being used: “the primary functions of a software process model are 
to determine the order of the stages involved in software development and 
evolution and to establish the transition criteria for progressing from one 
stage to the next” (Boehm, 1988, p.61). 

As already mentioned, design and evaluation are two examples of 
these stages. Yet, this has not always been the case. One of the earliest 
process models is the code-and-fix model (Boehm, 1988). It contained 
only two steps: write code and fix code. “The order of the steps was to do 
some coding first and to think about the requirements, design, test, and 
maintenance later” (Boehm, 1988, p.62). The problems that came along 
with it were poor structures, a mismatch to user needs, and expensive 
fixes as they typical resulted in also fixing the overall suboptimal structure. 
This led to the development of staged models like the waterfall model 
(Royce, 1970) (cf. Figure 4-1). It uses a set of linear sequential stages, 
where each stage depends on the deliverables of the previous one. Every 
stage corresponds to a specialized activity like analysis, design or testing. 
Although Royce described this process model as one of the first, he never 
used the term “waterfall model” himself (Royce, 1970). 
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Yet, also the waterfall model has its problems: the requirements ana- 
lyzed in the beginning turn out to change over time as more is learned 
about a topic or when users discover their real needs once a first version 
of the resulting product or service becomes available. These requirement 
changes thus become a risk to a project’s success. 

Therefore, process models evolved further into iterative, incremental 
models (also referred to as evolutionary models (O Regan, 2017)) like the 
spiral model (Boehm, 1988) (cf. Figure 4-2). 


“The major distinguishing feature of the spiral model is that it creates a risk-driven ap- 


Figure 4-1. The stages of the waterfall model (Royce, 1970). 


proach to the software development process rather than a primarily document-driven or 
code-driven process. It incorporates many of the strengths of other models [like the code- 
and-fix model, and the waterfall model] and resolves many of their difficulties” (Boehm, 
1988, p.61). 
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Figure 4-2. Spiral model (Boehm, 1988). 


Iterative and incremental process models are the basis for mostly any 
type of human-centered design approach in HCI or CSCW (Gross & Koch, 
2007). In the following, this chapter takes a look at how single-user appli- 
cations and then more specifically groupware is designed and evaluated 
this way. It outlines existing common procedures and then dives deeper 
into the area affecting the support functions of awareness and coordina- 
tion. For these especially the limitations of current approaches are dis- 
cussed. This chapter, as the previous ones, concludes with a critical reflec- 
tion on the overall topic in the context of awareness and coordination be- 
fore summarizing its key insights. 
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4.1 From Participatory to Human-Centered Design 


HCI, CSCW and this thesis focus on systems intended to be used by hu- 
mans, i.e., socio-technical systems (cf. section 1.1). “The design of single- 
user applications translates users’ tasks and needs into a functional de- 
scription which directs the overall design and development of the applica- 
tion” (Ehrlich, 1999, p.2). Especially HCI developed two answers to the 
aforementioned risk of changing user requirements: 

1. The utilization of an iterative and incremental process model de- 
veloping one group of requirements at a time while checking the 
validity of others. 

2. The active engagement of users in the development process to re- 
gard their insights and feedback as early as possible. 


While the first aspect was already known from the spiral model, the 
second part evolved from a procedure initially outside of HCI known as 
participatory design (Muller & Kuhn, 1993; Ross et al., 1995): 


“The field of participatory design (PD) spans a rich diversity of theories, practices, anal- 
yses, and actions, with the goal of working directly with users (and other stakeholders) 
in the design of social systems including computer systems that are part of human work” 
(Muller & Kuhn, 1993, p.24). 


PD is also referred to as the Scandinavian approach or as cooperative 
design or co-design (Preece et al., 2019). It is an overarching design philos- 
ophy not only limited to the design of IT technologies (Gregory, 2003). 

In the area of HCI (especially from the angle of (again) cognitive psy- 
chology) participatory design evolved into its more IT-related counterpart 
of user-centered design (UCD). Norman (1988) describes it in his well- 
known book The Psychology of Everyday Things (POET) as “a philosophy 
based on the needs and interests of the user, with an emphasis on making 
products usable and understandable” (Norman, 1988, p.188). 

With the third edition of his book and a slightly changed title to The 
Design of Everyday Things (DOET), the approach also evolved to its current 
incarnation of human-centered design (HCD): “Human-centered design 
(HCD) has emerged since the first edition, partially inspired by that book. 
(...) although looking back, we see that the entire book [POET] was about 
HCD” (Norman, 2013, p. xv). Further, he defines HCD as “an approach 


Design & Evaluation of Cooperative Applications | 125 


that puts human needs, capabilities, and behavior first, then designs to 
accommodate those needs, capabilities, and ways of behaving” (Norman, 
2013, p.8). The subtle difference in names indicates that HCD is not only 
limited to users but also regards other stakeholders. Further, HCD intro- 
duced another concept which goes beyond the “understandable and usa- 
ble”: the user experience (UX) (Norman, 2013, p. xiii). It can briefly be 
defined as “a user’s perception and responses that result from the use 
and/or anticipated use of an interactive system” (UXQB, 2020, p.12). 

In HCI it is nowadays commonplace to apply the human-centered de- 
sign process (or one that encompasses the same steps but uses a different 
name like the interaction design process in Dix et al. (2004) or Preece et 
al. (2019)). Today, the process is specified by the norm ISO 9241-210 (ISO, 
2019) which was released in its second edition in 2019. Briefly, its stages 
encompass the understanding and description of the context of use, spec- 
ifying the user requirements, designing a solution on the basis of those 
requirements, and the evaluation of the design (cf. Figure 4-3). 


Plan the human-centred 
design process 


Understand and specify 
the context of use 


Designed solution meets 
user requirements 7 


5 - 
Iterate, where appropriate 
Igor 

l4 
L 
Evaluate the designs against 
requirements \ 


Specify the user 
requirements 


\ 
\ 
1 


Yy 


Produce design solutions to meet 
user requirements 


Figure 4-3. Human-centered design process as defined in ISO 9241-210 (ISO, 2019). 
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However, this model does not indicate what it means by design or eval- 
uation, i.e., it is agnostic of the methods eventually used — and there are 
many possible ones to fill these gaps (to be shown in section 4.3). This in 
particular is the case where the process is expected to be different between 
HCI and CSCW. Further, its depiction (cf. Figure 4-3) understates the 
model’s incremental nature. There are early stages, phases, or increments 
and later ones that truly affect the techniques and methods to be used, i.e., 
there is a dependency between the stage of the process and the applicabil- 
ity of a certain method. 

The human-centered design process can be applied straight forward in 
commercial software development but also as a research approach. The 
contribution section of this thesis (cf. section 1.4) described HCI as an 
example scientific discipline for use-inspired basic research. “Science is 
essentially a problem-solving activity” (Laudan, 1978, p.11) and the prob- 
lems in science are typically resolved using the scientific method 
(Encyclopaedia Britannica, 2019) that includes the following procedure 
(cf. Table 4-1): 


Table 4-1. Steps of the scientific method. 


Step # Description Area 

1 Define a question Hypothesis generation 
2 Gather information 

3 Formulate hypothesis 

4 Test experiment Hypothesis validation 
5 Analyze data 

6 Interpret data and reformulate hypothesis 


Steps 1-3 are also referred to as hypothesis generation whereas steps 
4-6 belong to hypothesis validation. 

Adding humans to the picture for the case of HCI, the scientific 
method resembles the human-centered design process to a very high de- 
gree. The problems to be solved in HCI (and CSCW) (cf. section1.2) are 
typically framed as design challenges (open question — how can we design 
an easy to use application?) or design tensions (either-or question - is it 
better to either use recognition or recall?) (Tatar, 2007). HCI as a field for 
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use-inspired basic research (Stokes, 1997) allows not only to develop fully 
functional systems but to create prototypes of partial or isolated setups or 
trials to understand the basic working of a certain research aspect or con- 
struct (e.g., awareness support) or mechanism. The very same accounts 
for CSCW. 


4.2 The CSCW Design Challenge — A Wicked Problem 


Being an iterative and at the same time incremental process for designing 
interactive systems in general, the human-centered design process is also 
eligible for the development of cooperative systems. However, creating 
CSCW applications differs notably from developing single-user applica- 
tions. 


“In contrast to single-user applications which support peoples’ tasks, groupware sup- 
ports peoples’ work. Tasks are often explicit, observable and concrete. Work is often tacit, 
invisible and amorphous. The challenge in developing a groupware application lies in 
understanding, explicating and then supporting the invisible work” (Ehrlich, 1999, p.1). 


Neale et al. (2004) consider it “a paradigm shift toward socially cen- 
tered design from past eras of system-centered and user-centered design” 
(Neale et al., 2004, p.113; Stanney & Maxey, 1997). At a minimum, differ- 
ent and additional requirements need to be considered when designing 
CSCW applications (Wilson, 1991): 

e Individual human characteristics (e.g., communication pat- 

terns) !6 

e Organizational aspects (e.g., organizational structure) 

e Design of cooperative work together with the users, including sup- 

port functions 

e Group dynamics and processes 


Ehrlich (1999) suggests a phased approach consisting of the steps re- 
quirements, design and deployment, yet, leaving out evaluations (Ehrlich, 
1999). To understand work practices, she and others suggests the set of 
ethnography, participatory design, and action research (Wulf & Rohde, 


16 Tt is intersting that cognitive capabilities are left out here. 
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1995). All of this should occur in an iterative process (Tang, 1991) or cycle 
of design (Dourish, 1995). 

However, the design and evaluation of CSCW applications often failed 
mostly due to the following reasons (Grudin, 1988, 1994b; Lynne & 
Connolly, 1990; Bullen & Bennett, 1990): 


1. 


The disparity between those who benefit from an application and 
those who must do additional work to support it. 

The breakdown of intuitive decision-making. 

Underestimated difficulty of evaluating CSCW applications, pre- 
vents learning from experience. 


Later this list was extended to the overall number of eight challenges 
(Grudin, 1994b) with the additional ones being: 


4. 


Critical mass and prisoner’s dilemma problems. Groupware is 
only useful to all when reaching a critical mass or it may fail be- 
cause it is never of use to anyone. 

Disruption of social processes. Groupware may require activities 
that violate social norms and standards in the area of its applica- 
tion. 

Exception handling. Group activities are more complex making 
the exception handling also more complex. 

Unobtrusive accessibility. “Features that support group processes 
are used relatively infrequently, requiring unobtrusive accessibil- 
ity and integration with more heavily used features” (ibid., p.97) 
Adoption process. CSCW applications require a more careful in- 
troduction in the workplace as more individuals need to agree to 
the new style of working. 


Cockburn and Jones (1995) generalize the above into three levels of 
failure: system-use, system-design, and system-evaluation. In addition to 
that, they present four groupware design principles to avoid the above 
mentioned problems altogether (Cockburn & Jones, 1995): 
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1. Maximize personal acceptance (to encourage individuals to adopt 
new systems) 

2. Minimize requirements (reduce disparity of groupware cost and 
benefits as well es minimize efforts to collaborate). This also in- 
cludes effortless coordination. 

3. Minimize constraints (avoid inflexible and constraining style of 
use) (this is in line with the problem of explicit coordination sup- 
port, cf. section 3.3.5) 

4. Maximize external integration (this is what SLACK does, cf. section 
1) 


The early failures were the reason to use ethnographic studies exten- 
sively (cf. section 2.2.1). Due to the complexity of the issue demonstrated 
by all these failures, Fitzpatrick (1998) even considers the design of CSCW 
applications overall a true (design) challenge because “designing CSCW 
systems is a ‘wicked problem” (Fitzpatrick, 1998, p.11). A wicked problem 
is a problem that is difficult or impossible to solve because of incomplete, 
contradictory, and changing requirements that are often difficult to recog- 
nize (Rittel & Webber, 1973). In section 3.1.4 about problem-solving the 
same was also referred to as an ill-defined problem. It stands for an idea or 
problem that cannot be fixed or where there is no single solution to the 
problem. The use of the term ‘wicked’ denotes resistance to resolution, 
rather than evil (Rittel & Webber, 1973). Wicked problems require an on- 
going iterative dialogue between problem and solution to understand ei- 
ther. This corresponds to cycles of design and evaluation. Requirements 
change rapidly. Each step in the design might change the possible usage 
of the solution: 


“The design of new technology is always an intervention into an ongoing world of activ- 
ity. It alters what is already going on - the everyday practices and concerns of a commu- 
nity of people — and leads to a resettling into new practices, which in turn create new 
future design possibilities” (Flores et al., 1988, p.154). 


Therefore, CSCW research is indicative of an evolutionary human-cen- 
tered design process. Having chosen one to address the wicked problem 
of CSCW design and evaluation now raises the question of how to equip 
that process for the use in the context of CSCW. Plowman et al. (1995) 
“have constructed a diagrammic overview to show the various ways in 
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which work studies have been used to inform system design and facilitate 
the implementation of CSCW in organisations” (Plowman et al., 1995, 
p.319). Their overview (cf. Figure 4-4) is divided into three merging phases 
(initial research & implications, design & change phase, evaluation & de- 
velopment), each showing the kinds of research activities that take place 
at that particular stage in the design cycle as well as their potential out- 
comes. Overall, the process is in line with the human-centered design pro- 
cess, though the naming of the phases appears a little odd. 
1. Initial Research & 
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Figure 4-4. Role of studies in CSCW design and implementation (Plowman et al., 1995). 
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Their overview is also one of the first assigning methods to process 
stages in the context of CSCW. Yet, they make no further mention of early 
and late design cycles. 
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More recent and stemming from the angle of wicked problems, De- 
sign Thinking (DT) is another approach to cope with wicked problems 
(Buchanan, 1992; Dolata & Schwabe, 2016): 

“Many see the primary impact of Design Thinking in the area of industrial innovation. 

Given the engineering background of the methodology, this is definitely the most 

straightforward approach, with its practice-oriented nature. We claim, however, that De- 

sign Thinking - defined as the mindset as well as the toolset — can significantly contrib- 


ute to the success of academic research in the information systems area” (Dolata & 
Schwabe, 2016, p.67). 


Design Thinking is a human-centered design approach that also facil- 
itates multidisciplinary teams and collaborative workspaces (GDTA, 
2018). There are multiple variations of its design process (cf. Figure 4-5) 
describing 5 to 7 steps as part of an evolutionary model going back and 
forth between design and evaluation or hypothesis generation and valida- 
tion. The model is also agnostic of the actual methods being used in each 
phase making it ready-for-use also in the area of CSCW to tackle its own 
wicked problems. 
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Figure 4-5. 5 step design process of Design Thinking (Hasso Plattner Institute, 2010). 
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4.3 


Evaluation in General 


This section focuses especially on the evaluation (or test) stage as part of 
an iterative and incremental process. It is one of the core steps of many 
development processes but in particular of the human-centered design 
process. Yet, in most cases this happens without defining any concrete 
method on how to evaluate anything. 

As part oftheir research, Twidale et al. (1994) came up with 9 different 
definitions of evaluation (a term they use synonymously with the term 
assessment) (Twidale et al., 1994, p.442): 


1. 


An assessment of the overall effectiveness of a piece of software, 
ideally yielding a numeric measure by which informed cost-ben- 
efit analysis of purchasing decisions can be made. 

An assessment of the degree to which the software fulfills its spec- 
ification in terms of functionality, speed size or whatever 
measures were prespecified. 

An assessment of whether the software fulfills the purpose for 
which it was intended. 

An assessment of whether the ideas embodied in the software 
have proven to be superior to an alternative, where that alterna- 
tive is frequently the traditional solution to the problem ad- 
dressed. 

An assessment of whether the money allocated to a research pro- 
ject has been productively used, yielding useful generalizable re- 
sults. 

An assessment of whether the software proves acceptable to the 
intended end-user. 

An assessment of whether end-users continue to use it in their 
normal work. 

An assessment of where the software fails to perform as desired 
or as is now seen to be desirable. 

An assessment of the relative importance of the inadequacies of 
the software. 


A valid number 10 would have been the validation of research hypoth- 
eses. The items above describe ‘what’ and partially ‘why’ something is 
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examined as part of an evaluation. According to Dix et al. (2004, p.319) 
evaluation has three main goals: 
1. To assess the extent and the accessibility of the system’s function- 
ality, 
2. to assess the users’ experience of the interaction, and 
3. toidentify any specific problems with the system. 


Once the goal is clear, an appropriate evaluation procedure needs to be 
determined. Most testing done in HCI is usability testing. In general, 
“there are generally two types of usability tests: finding and fixing usability 
problems (formative evaluation) and describing the usability of an appli- 
cation using metrics (summative evaluation)” (Sauro & Lewis, 2012, p.10). 

Figure 4-6 depicts the ‘how’, i.e., the way something can be evaluated, 
by identifying different evaluation types. Pinelle and Gutwin (2000) dis- 
tinguish four types depending on the setting and degree of manipulation 
(McGrath, 1995; Pinelle & Gutwin, 2000). 


Manipulation 


Rigorous Minimal/None 


Setting 
Controlled Lab experiment Exploratory 


Figure 4-6. Evaluation types (Pinelle & Gutwin, 2000). 


Preece et al. (2019) use a slightly different approach: “We classify eval- 
uations into three broad categories, depending on the setting, user in- 
volvement and level of control” (p.500). They distinguish: 

e Controlled settings directly involving users (usability labs, re- 
search labs): User activities are controlled to validate hypotheses 
and measure or observe certain behavior. The main methods are 
usability testing and experiments. 
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Natural settings involving users (e.g., online communities, field 
studies): There is little or no control of the users’ activities to de- 
termine how the product would be used in the real world. The 
main method used is field studies. 

Any setting not directly involving users: inspections, heuristics, 
walk-throughs as well as models and their analysis. 


All of the above shows that there are many different dimensions to 
evaluation and its respective techniques: 


Point in time (of the development cycle) when it is to be used 
(from early to late) (Pinelle & Gutwin, 2000) 

Effort and cost caused by the approach (Baker et al., 2001) 
Formative testing (finding and fixing usability problems) or sum- 
mative testing (describing the usability of an application overall) 
(Sauro & Lewis, 2012) 

User participation or expert-driven inspections (Molich & Nielsen, 
1990) 

Natural or artificial settings (McGrath, 1995) 

Degree of manipulation (McGrath, 1995) 

Degree of disruption during evaluation (McGrath, 1993; McGrath 
et al., 2000) 

Evaluation of single aspects (e.g., satisfaction) or multiple/com- 
bined aspects as in usability (i.e., effectiveness, efficiency, and sat- 
isfaction) 

Single instance or comparative setups (A/B or split testing) (Sauro 
& Lewis, 2012) 

Targeting a user’s attitude or behavior (Nielsen, 2001) 

Resulting in quantitative or qualitative data (Sauro & Lewis, 2012) 
To validate or generate hypotheses (cf. Table 4-1 in section 4.1) 


There are many pros and cons of each category. Lab-based studies are 
good at revealing usability problems, but they are poor capturing the con- 
text-of-use; field studies are good at demonstrating how people use tech- 
nologies in their intended setting, but they are often time consuming and 
more difficult to conduct (Rogers et al., 2013). Modeling and predicting 
approaches are relatively quick to perform, but they can miss 
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unpredictable usability problems and subtle aspects ofthe user experience 
(cf. section 4.1). Similarly, analytics are good for tracking the use of a web- 
site but are inappropriate for finding out how users feel about a new color 
scheme or why they behave as they do. The following takes a quick look at 
the most common evaluation types: experiments, field studies and inspec- 
tions. 
An experiment is a procedure carried out to support, refute, or validate 
a hypothesis (cf. scientific method in section 4.1). They may take place in 
the wild or in the lab and they prefer highly controlled settings to control 
independent and dependent variables in order to judge on cause and ef- 
fect. They are run for the following reasons (Dean et al., 2017): 
1. To determine the principal causes of variation in a measured re- 
sponse. 
2. To find the conditions that give rise to a maximum or minimum 
response. 
3. To compare the responses achieved at different settings of control- 
lable variables. 
4. To obtain a mathematical model in order to predict future re- 
sponses. 


“Observations can be collected from observational studies as well as 
from experiments, but only an experiment allows conclusions to be drawn 
about cause and effect” (ibid, p.1). Experiments especially help with vali- 
dating hypotheses. 

Ethnographic field studies (cf. section 2.2.1) have been the center piece 
in CSCW research especially allowing the following (Blythin et al., 1997): 

e Find new ways for developing new products 

e Evaluate existing socio-technical systems 

e Find ideas for the specification of designs 


Field study-based methods are mostly used for hypotheses generation 
as they are mostly based on qualitative measures. Yet, they have been crit- 
icized for being less appropriate for producing design solutions (cf. sec- 
tion 2.4.4). 
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Last but not least, inspection is a generic term for a set of methods 
where evaluators (experts, not the users) inspect an application’s user in- 
terface to spot usability problems: 

e Heuristic evaluation (Nielsen & Molich, 1990) 

e Cognitive Walkthrough (Polson et al., 1992) 

e Formal usability inspection (Nielsen, 1995) 

e = Pluralistic walkthroughs (Nielsen, 1995) 

e Feature inspection (Nielsen, 1995) 

e Consistency inspection (Nielsen, 1995) 

e Standards inspection (Nielsen, 1995) 


Overall, the evaluation method used in the end strongly depends on 
factors like the research goal, type and subject. Additionally, in terms of 
development cycles all of them will not work at any time! 

For instance, when it comes to the evaluation of the user interface 
there are currently four possible ways (Nielsen & Mack, 1994; Nielsen, 
1995): 

1. Automatically: usability measures computed by running a user in- 

terface specification through a program, e.g., the number of clicks 
a user needs to fulfill a task. 

2. Empirically: usability assessed by testing the interface with real 

users. 

3. Formally: using exact models and formulas to calculate usability 

measures, as with GOMS (Card et al., 1983). 

4. Informally: based on rules of thumb and the general skill and ex- 

perience of the evaluators. 


However, “under the current state of the art, automatic methods do 
not work and formal methods are very difficult to apply and do not scale 
up well to handle larger user interfaces” (Nielsen, 1995, p.377). That is 
why empirical and informal methods are most likely to be used. Due to 
some of the disadvantages (e.g., cost and time) and the need of something 
without a high degree of precision led to the development of discount us- 
ability engineering techniques (Nielsen, 1994). All of the above evaluation 
techniques focus on the interaction between humans and computers. 


Who partiipates with whom in what 


Designers participate in user's worlds 


Users directly participate in design activities 
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They let the evaluator judge on the interaction in three dimensions that 
comprise the definition of usability (ISO, 2019): 

1. Efficiency 

2. Effectiveness 

3. Satisfaction 


When to use which user experience method, depends on the following 
factors: 
1. Stage of development and who participates (cf. Figure 4-7) 
2. User information intentionally provided or given off (attitude/be- 
havior) versus data requirements (quantitative/qualitative) (cf. 
Figure 4-8) 
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Figure 4-7. Methods for participatory design (Muller & Kuhn, 1993). 
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Figure 4-8. User research methods (Rohrer, 2014). 


4.4 Evaluation in CSCW 


Evaluation also plays an important role in CSCW. “Few practitioners in 
CSCW would wish to contest the importance of evaluation work. In prin- 
ciple, evaluation should be a significant check of a system’s capacity to 
deliver what is required of it” (Twidale et al., 1994, p.441). Yet, already 
more than 20 years ago, CSCW researchers suspected that evaluations are 
not necessarily trivial: “Our problems in finding an acceptable basis for 
‘validating’ the system led us to suggest there is a pressing need for the 
reappraisal of evaluation philosophies and techniques” (Twidale et al., 
1994, p.441). 
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The difficulty with the evaluation of groupware has been known from 
the very early days of CSCW(Grudin, 1988; Ross et al., 1995) (cf. section 
4.2). One ofthe reasons is the tight entanglement of social and technical 
aspects which is typical for socio-technical system (cf. section 1.1). The 
predominant assumption is that the technical cannot be tested without 
the social (Gross & Koch, 2007). “The evaluation of CSCW applications 
requires a very different approach, based on the methodologies of social 
psychology and anthropology” (Grudin, 1988, p.87). Researchers have 
constantly been facing the intellectual challenge of the gap of what must 
be supported socially and what can be support technically (Ackerman, 
2000). Further, it became quickly obvious that CSCW evaluations are dif- 
ferent from those conducted in HCI. 


“Much evaluation of the past was concerned with the cognitive functioning of a single 
user sitting alone in front of a computer display. Users were modeled as vigilant, task- 
oriented workers operating in relatively narrow contexts over short time periods without 
regard to their broader functioning as social members of larger groups and communi- 
ties” (Neale et al., 2004, p.112). 


But besides the initial differences, the problem with evaluation in 
CSCW is complex. 


“CSCW evaluation has been, in general, more broad in nature, but it often has been ill- 
defined, time consuming, labor intensive, difficult to implement, difficult to interpret, 
and largely ineffective at producing formative data that is needed if groupware applica- 
tions are to succeed” (ibid, p.112). 


Neale et al. (2004) identified the following major issues: 
1. Logistics of data collection in distributed settings 

2. Number and complexity of variables 

3. Validating re-engineering group work 
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In the field of CSCW, evaluation frameworks fall into three different 
categories (Neale et al., 2004, p.114): 

1. Methodology-oriented frameworks: they describe the types of ex- 
periments and methodologies available to CSCW researchers. 
They provide an overview but no further help on when to choose 
which method. 

2. Conceptual frameworks: describe the group factors that need to be 
considered during evaluation but they do not describe the meth- 
odology on how to achieve that. 

3. Concept-oriented frameworks: focus on specific aspects of group 
behaviors or concepts, such as communication or coordination. 
They are more limited but offer specific help to focus on isolated 
aspects of group interactions. 


It has always been an issue whether groupware can be evaluated only 
using ethnographic studies (cf. section 2.2.1). Other researchers “believe 
that it is more practical to evaluate groupware through usability inspection 
methods” (Steves et al., 2001, p.125). For the most part, these inspection 
methods have been translated and adapted from the context of HCI: 

e Basic inspection (Steves et al., 2001) 

e Groupware Walkthrough (Pinelle & Gutwin, 2002) 

e Heuristic evaluation (Baker et al., 2002) 

e Collaboration Usability Analysis (CUA) (Pinelle et al., 2003) 

e Lab simulation (Humphries et al., 2004) 


On the other end, field studies have been criticized for being less ap- 
propriate for producing design solutions (Neale et al., 2004; Plowman et 
al., 1995). Further they require a functioning groupware system running 
in the actual workplace (Neale et al., 2004). Therefore they do not integrate 
well in the iterative design process inherent to interactive system design 
(Dix et al., 2004) (cf. section 4.1) as they cannot be deployed at the early 
design stages. At the evaluation stage most studies focus on groupware 
usability (Wainer & Barsottini, 2007). Groupware usability is roughly de- 
fined as the “degree to which a groupware system supports the activity of 
collaboration” (Gutwin & Greenberg, 1999, p.247). This includes taskwork 
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and teamwork (cf. section 2.2.2). They typically employ indirect measures 
ofthe following (Olson et al., 1995; Olson et al., 1992): 

e Product measures: (taskwork elements) measure aspects of the 
outcome of a task. Assumption: usability positively influences the 
group’s success completing the task. 

e Process measures: (teamwork elements) look for patterns in be- 
havioral and verbal activity during a collaborative session to judge 
on effectiveness and efficiency (observation, visual recording). 

e Satisfaction measures: (both teamwork and taskwork) subjective 
experience of the individual user with the groupware system 
(questionnaires, interviews). 


As part of their groupware usability testing Gutwin and Greenberg 
(1999) particularly selected the following five measures (p.243): 

1. Task completion times (task work): group activities relate to per- 
formance, effectiveness 

2. Communication efficiency: number of words spoken related to 
subject (a coordination practice referred to as articulation work in 
section 3.2.2) 

3. Participants perceived effort: activities of collaboration 

4. Overall preference: satisfaction measure, assumes the relation- 
ship between usability and preference; users prefer a system that 
better supports their collaborative activities 

5. Strategy use: how groups choose to solve the task 


4.5 Evaluation of Awareness & Coordination Support 


When it comes to the evaluation of awareness and coordination support, 
especially later in terms of effort, there is only little that was developed in 
the area CSCW. The major part stems from the study on human-factors. 
Overall, it is not a trivial task: 
“Because SA is an internalized mental construct, creating measures to adequately assess 
and describe it is not an easy task. Metrics of SA generally approach the issue either by 


inferring SA from other constructs that are easier to assess, or by attempting to obtain a 
direct assessment of the operator’s SA” (Endsley & Jones, 2004, p.259). 


142 | Design & Evaluation of Cooperative Applications 


This especially underlines the need to develop new evaluation tech- 
niques to be used prior to real workplace evaluations in order to make 
them more practical in terms oftime and cost and to eliminate many prob- 
lems early on thus improving the overall efficiency of evaluation (Pinelle 
& Gutwin, 2000). However, this cannot be reduced to simply using dis- 
count techniques from HCI. 


“Within the field of human-computer interaction (HCI), many low-cost evaluation tech- 
niques have moved out of the research arena and into accepted practice (...) [yet, they] 
have problems when we try to apply them verbatim for the purpose of evaluating group- 
ware” (Baker et al. 2001, p.124). 


The mechanics of collaboration (Baker et al., 2001; Gutwin & Greenberg, 
2000; Pinelle et al., 2003) are a methodology-oriented framework (Neale et 
al., 2004) suggesting an evaluation scheme that is well apt to “occupy a 
middle ground between brittle experimental techniques and time-con- 
suming field techniques, where they will provide the kind of formative 
information valuable in an iterative groupware development process” 
(Gutwin & Greenberg, 2000, p.120). The mechanics of collaboration are 
the low-level actions (communication, coordination, planning, monitor- 
ing, assistance and protection) and interactions that must take place when 
completing a task as a team. Their main characteristic (and advantage) of 
these mechanics: they are independent of the social context! This fits the 
picture previously drawn in this thesis about awareness and coordination 
support (cf. sections 2.4.7 and 3.2.3). The downside of this framework is 
that it lacks the concrete means of measurement for each of the mechan- 
ics. “Aside from gross descriptions like laboratory or field studies, it is 
difficult to determine how these factors should be studied, and it is diffi- 
cult to determine what methods are best suited to which factors” (Neale et 
al., 2004, p.114). 

Therefore, starting with CSCW and having established a background 
on evaluation, the next step is to take a look at examples of evaluations in 
CSCW dealing with awareness or coordination to see whether and how 
other researchers have approached the issue and to understand where 
things might have gone wrong in the past. In fact, there are many evalua- 
tion approaches and techniques in CSCW originating from various re- 
search areas like interviews, questionnaires (social psychology), ethno- 
graphic studies (sociology), conversation analysis (ethnomethodology) etc. 
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(Ross et al., 1995). Pinelle and Gutwin (2000) conducted a review espe- 
cially focusing on groupware evaluations: 


“We reviewed all papers from the ACM CSCW conference (1990-1998) that introduced 
or evaluated a groupware system. Forty-five papers were included in the review. The 
main findings are that almost one-third ofthe groupware systems were not evaluated in 
any formal way, that only about one-quarter ofthe articles included evaluations in a real- 
world setting, and that a wide variety of evaluation techniques are in use” (p.86). 


Most notably: 7 out ofthe 10 most severe usability problems of group- 
ware related to coordination and awareness. In order to briefly outline 
what researchers actually did during their evaluations the following se- 
lects a small sample of publications with different approaches. The goal 
here is not to compile an exhaustive list of all evaluations but to select 
different approaches that were actually used for the purpose of evaluating 
awareness and/or coordination. For each of these publications it is 
checked if the aspect of effort is addressed, if a comparative approach is 
used, and if researchers regarded awareness or coordination characteris- 
tics (ephemeral, subconscious etc.) in their setups. 

The first publication is from Dourish and Bellotti (1992) who showed 
that awareness in group editors has positive effects on the coordination in 
work teams. They used an experimental approach conducting a lab study 
with a shared text editor. The limitations of the editor imposed further 
restrictions to the settings. They used a task-based evaluation and ob- 
served their participants as the sessions were videotaped. Further, they 
conducted a post-task interview. The study rather served the purpose to 
get a general understanding of a cooperative context and the role of aware- 
ness in it in terms of qualitative feedback. The goal was not to improve 
awareness as it was not operationalized further for measurement. There 
was no quantitative verification or comparative approach. 

In another publication, Gutwin et al. (1996) conducted a usability study 
(combined aspects as compared to Gross (2013)) of awareness widgets. 
They used post-task questionnaires and interviews, that were audiotaped. 
Awareness itself was not operationalized. Participants were asked directly 
for their feedback and consent regarding the widgets. Thus, the authors 
provided an insight on their users’ attitude which is valuable to build hy- 
potheses — that need to verified quantitatively. It was not proven by any 
numbers that their users could coordinate better using a particular widget, 
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indicating a certain implementation’s superiority to others (to resolve a 
certain design tension). 

In another study, Convertino et al. (2004) used a new approach of field 
studies informing the design of lab experiments for the case of activity 
awareness. They used a task-driven approach applying the Think Aloud 
technique (Nielsen, 1994) to learn what users were thinking while com- 
pleting a task thus imposing an even higher load on the working memory. 
They especially focused on awareness breakdowns using multiple session 
in between which they requested informal feedback (contextual inquiry). 
Further, they used a post-task questionnaire. Two researchers subse- 
quently assessed the level of awareness post-task by judging on the degree 
of inconsistencies being noticed (i.e., expert judgement). 

The approach reported by Antunes et al., (2014, 2010) to review the 
quality of awareness support in collaborative applications is based on a 
checklist comprised of 54 design elements and six awareness types. The 
list is created by experts to be provided to developers as a guide in the early 
stages of development. Yet, they did not operationalize awareness or co- 
ordination in terms of effort. They do not measure the user’s effort but 
rate the application based on expert knowledge. Further, they did not pro- 
vide explicit before and after results of an application checked using their 
method. 

All of the above used a mixture of multiple evaluation techniques. 
Most of them used either high-fidelity prototypes or fully developed appli- 
cations as test objects, that is, they were not at the very beginning of the 
development process. Only one had a comparative setup (not in terms of 
before/after but A/B). None of the above focused on efforts imposed by 
coordination or awareness. The applied techniques were mostly borrowed 
from HCI, some with slight adaptations to be used in the context of group- 
ware. Most of them do not specifically regard or even ignored the charac- 
teristics of awareness or coordination as outlined above. For instance, 
post-task questionnaires and interviews tend to query the users’ attitude 
about issues they have not consciously thought about at a point in time 
when the knowledge is typically gone. 

Turning to human-factors, reveals a more advanced approach in terms 
evaluating situation awareness (SA). As described earlier, researchers de- 
veloped a model for situation awareness (cf. section 2.1.2) and the great 
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advantage when having this kind of model is, that it is a lot easier to de- 
velop measurement and evaluation approaches based on it. “In general, 
evaluation metrics can assess user processes, cognitive states, and perfor- 
mance at different stages” (Endsley & Jones, 2004) (cf. Figure 4-9). 


Direct 
measures 


Process 
measures 


Behavior and performance measures 


Information Situation 
assessment System m> Decisions = Performance 
awareness 
processes 
Individual Strategies Doctrine Fatigue System capabilities 
moderating Skills Procedures Training 
factors: Knowledge Training Skills 
Abilities Personality factors Expertise 


Figure 4-9. Areas of evaluation (Endsley & Jones, 2004). 


Further, researchers believe that “direct and objective measurement of 
SA is the best way to approach the evaluation of a system design” (Endsley 
& Jones, 2004, p.260). For this to occur they suggest the following 
measures: 
e Subjective measures: qualitative data, users rating their SA on a 
certain scale revealing that something happens and why 
e Objective measures, i.e., collecting data from users and their per- 
ception of the context compared to what is actually happening in 
that context. In detail there are: 
o Measures of effectiveness 
o Measures of performance 
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e Process measures - including eye tracking, information acquisi- 
tion, and communication analysis 
e Workload measures 
o Physiological measures (EEG/ECG) 
o Subjective measures 
o Performance measures including task error measures 
and measures of spare capacity using concurrent second- 
ary tasks 


When it comes to workload measures, there is an interesting observa- 
tion to be made as far as the measurement outside the context of cooper- 
ative work is concerned: 

“One of the most widely used techniques to measure workload is the secondary task. 

This technique requires an operator to perform the primary task with that task’s speci- 

fied requirements and to use any spare attention or capacity to perform a secondary task. 


The decrease in performance of the secondary task is operationally defined as a measure 
of workload” (Gawron, 2008, p.94). 


The above covers the theory. Talking about the practical measurement 
of situation awareness two well-known techniques from human factors 
are the Situational Awareness Rating Technique (SART) (Taylor, 1990; 
Taylor et al., 1998; Endsley et al., 1991) and the Situation Awareness Global 
Assessment Technique (SAGAT) (Endsley et al., 1998). 

SART is an approach using subjective metrics. It assumes, that sub- 
jects “use some understanding of situations in making decisions, that this 
understanding is available to consciousness and that it can readily be 
made explicit and quantifiable” (Taylor, 1990). The method clusters 10 
constructs into three broad domains: 

1. Attentional demand: situation stability, variability and complexity 

2. Attentional supply: arousal, spare mental capacity, concentration 

and division of attention 

3. Understanding: information quantity and quality as well as famil- 

larity 


Especially 1 and 2 are associated with workload which is seen as an 
integral part of SA (Taylor, 1990). 
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The Situation Awareness Global Assessment Technique (SAGAT) 
(Endsley et al., 1998) is an objective metric used across a variety of do- 
mains like air traffic control, power plant operations, teleoperations, driv- 
ing and military operations. SAGAT seeks to assess a subject’s SA by ask- 
ing questions and comparing the results to reality. This is done during a 
simulation exercise where 


“at randomly selected intervals, the simulation activity is briefly halted, the displays are 
blanked, and a battery of queries is administered to the participant (...) Data collected by 
the simulation computer and by subject matter experts are used to score the participant’s 
responses as correct or incorrect based on what was actually happening in the scenario 
at that time” (Endsley & Jones, 2004, p.271). 


After answering the question, the simulation resumes. Major disad- 
vantages or limitations of the technique are that it requires a situation to 
be stoppable which is likely not the case in many real-world settings. An- 
other is the creation of appropriate questions that fit the environment. 

Objective measures like SAGAT typically face the following chal- 
lenges: 

e Appropriate timing for the queries; asking the subject in the end 
provides the subject’s SA at the point when the activity ended (this 
is also the problem with subjective questionnaires). 

e Appropriate questions to ask related to the subject’s task of inter- 
est. 

e The questions must not alter the subject’s SA by asking for cir- 
cumstances the subject is not aware of. 

e Creating questions that cover the entire range of relevant issues to 
SA. 


The choice of metrics for an evaluation depends largely on the goals of 
the evaluation, the characteristics of the medium available for testing, and 
the expertise of the personnel performing the study. “How well a meas- 
urement technique addresses these challenges provides a benchmark for 
assessing the utility of the tool” (Endsley & Jones, 2004, p.283). 

Disadvantages of SARS and SAGAT are that they leave out coordina- 
tion and cooperative contexts completely. They require tremendous man- 
ual configuration and setup and ask users about awareness related items 
directly. 
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4.6 Challenges, Reflections and Criticism 


As in the previous chapters, there are some issues in this chapter worth 
discussing or for taking a second look. They are all organized in short sec- 
tions for easier reference later. 


4.6.1 The Toughest Question 


What is currently the best awareness support? This is a question that no 
CSCW researcher is currently able to answer. What is meant by “best”? 
Even a more relaxed variant thereof like “Is awareness support A better 
than B?” and “Why?” are equally tough. An earlier section talked about 
design tensions (cf. section 2.4.2), however, many ofthem stay unresolved 
as there are many hypotheses on how good or better awareness support 
might be like (i.e., the hypothesis generation was done) but there is hardly 
a method to validate these hypotheses. And the problem is not what is 
meant by good, better or best. For instance, “is it better to provide aware- 
ness cues inside the task screen or separate?” is a simple question which 
lacks a quantitative basis to answer it. Most evaluations use the usability 
criteria of effectiveness, efficiency and satisfaction (Frøkjær et al., 2000; 
Gutwin & Greenberg, 2000). Yet, what especially is effectiveness and effi- 
ciency in terms of awareness and coordination support? Both are closely 
related to the issue of effortless coordination. There is an issue with eval- 
uation to be explained in greater detail in section 4.6.8. For now the argu- 
ment ought to show that more basic research is required (cf. section 
2.4.1), which is for the case of awareness and coordination use-inspired 
basic research (Stokes, 1997). However, this in turn requires methods that 
can be used for hypothesis validation. 


4.6.2 False Consensus 


Groupware designers, as many other designers, easily fall into the trap to 
make decisions for the user based on their own preference. What is bad 
practice for single-user application becomes worse for groupware: 


„Ihe design process fails because our intuitions are poor for multi-user applications — 
decision makers see the potential benefits for people similar to themselves, but don’t 
see the implications of the fact that extra work will be required of others” (Grudin, 1988, 
p.86). 
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Fora CSCW application many different characters from different per- 
spectives need to participate in the design process in order to achieve a 
successful design. If not makes CSCW applications easily susceptible to 
the false consensus effect (Ross et al., 1978): 


“False consensus exists when people’s won choices, attitudes, or beliefs bias their esti- 
mates of those of other people, leading them to view their own reactions as relatively 
common while viewing alternative reactions as relatively uncommon. False consensus 
is revealed when people making a particular choice consider this choice more common 
than do people making the opposite choice” (Kunda, 1999, p.397). 


4.6.3 The Problem with the Study of Groups 


If the problems with awareness and evaluation were not troubling 
enough, there comes even another one. In his reflection on the study of 
groups McGrath et al. (2000) draw the following conclusion: 


“By concentrating our empirical research (and our subsequent theoretical formulations) 
on studies with these features, we have denied ourselves the opportunity to envision 
groups in ways that more accurately reflect our own experience in groups — namely, that 
groups are complex, adaptive, and dynamic systems — and to find ways to incorporate 
such a viewpoint in our empirical and theoretical research” (p.97). 


They especially argue against laboratory studies as these study groups 
as simple systems isolated from their context as static entities without 
past, present or future. 

This thesis takes quite the opposite direction as awareness and coordi- 
nation mechanics allow to abstract their use from a particular context (cf. 
section 2.4.7 and section 3.2.3). It is hard to study social dynamics but, in 
this case, it is not the goal! 


4.6.4 Hypothesis Generation versus Hypothesis Validation 


There are methods that are primarily apt to make observations upon 
which hypotheses can be build. For these usually empirical studies yield- 
ing qualitative data are applied. Once observations to be checked are 
formed into hypotheses it is time to make a switch towards hypothesis 
validation. This step typically applies another set of methods either 
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yielding further qualitative data or quantitative data that can be statistically 
analyzed. 

Ethnographic studies are the best choice to make observations (cf. sec- 
tion2.2.1) but not a good choice to validate them as they usually do not 
yield to much quantitative data that can be used for validation. 


4.6.5 Don’t Listen to Users 


This claim is one of the most famous of Jakob Nielsen (Nielsen, 2001) 
when being asked for the number one rule in usability research. It strikes 
the issue of user behavior versus their attitude. Users simply do not do 
what they say from which Nielsen concludes that he does not listen to 
them but rather observes their actions. For instance, in the COORDINATOR 
system they found that the “’wonderful’ functions often cited by users 
were not actually used by those users” (Bullen & Bennett, 1990; 
Henninger, 1991). This also becomes important when evaluating an 
awareness or coordination support system. Asking users about their opin- 
ions or attitudes does not help: 

1. They actually might use the system in a different way then articu- 
lated. 

2. It does not matter what they say as it would refer to a secondary 
task that works mostly subconscious and unintentional (cf. section 
2.5). 

3. Especially asking users about awareness after a common task 
makes no sense as the knowledge was not only mostly subcon- 
scious but already gone and forgotten by that time anyway (cf. sec- 
tion 2.5). 

4. Targeting effortless coordination efforts cannot be measured this 
way. 


That means that all approaches evaluating awareness and/or coordina- 
tion support should find another way than asking users directly about it. 
4.6.6 Assessment versus Approval 


Another problem related to asking users or customers is the difference 
between a true evaluation and asking for approval. In 2009, the US 
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company Walmart asked its customers: “Would you like Walmart to be 
less cluttered?” The customers agreed. 


“So, Walmart cleared out space and reduced inventory and customer satisfaction shot 
up. However, same-store sales plummeted, by Phil Terry’s estimate, by $1.85 billion, 
and now Walmart has fired the team that put the idea into place and is spending hun- 
dreds of millions to undo what they spent hundreds of millions doing. But wait! Weren’t 
they listening to their customers? Why weren’t they rewarded? It was a costly mistake 
that required lots of overhaul and refurbishing. 15% of inventory was removed from the 
stores. End caps were slimmed. Shelves got shorter. Gone were the big pallets of stuff 
stacked in the middle of aisles. Ah, it was more clean and open, more like Target - the 
strategy was put into place by a former Target exec, who is now a former Walmart exec 
— and sales dove. How could this be?” (Popken, 2011) 


Walmart’s goal was to listen to its customers, however, they “came up 
with the answer first, then asked customers to agree to it” (Popken, 2011). 
While the customers liked the increase in space inside the Walmart stores, 
the approach completely ignored what mattered most to the company: its 
vast selection of cheap items to create revenue. Many experiments men- 
tioned earlier used questionnaires for awareness evaluations running the 
risk of easily falling into the trap of rather asking for approval than con- 
ducting a real assessment. 


4.6.7 Situated Action versus Artificial Environments 


Section 3.3.4 already discussed the cognitive versus the social science. “A 
factor contributing to the failure to learn from experience is the extreme 
difficulty of evaluating these applications (...) these complex applications 
introduce almost insurmountable obstacles to meaningful, generalizable 
analysis and evaluation” (Grudin, 1988, pp.85). This section continues the 
argument which extends to how designers should evaluate their work. For 
instance, a study (Suchman, 1983) examined how existing office technol- 
ogies were being designed in relation to how people actually work. She 
found that designers would be much better positioned to develop systems 
that could match the way people behave and use technology, if they began 
by considering the actual details of work. Suchman later underlined her 
arguments highlighting the inadequacy of basing the design of an inter- 
active system purely on an abstract user model, when she analyzed the 
use of a help system for a photocopier showing the impact of unique 


152 | Design & Evaluation of Cooperative Applications 


details of the situation at hand (Suchman, 1987). This quite old study still 
fires the discussion on what evaluation type is appropriate for CSCW ap- 
plications. 

On the other end, real life scenarios are complex, hard to reproduce 
and conclusions to be drawn too specific to be generalized. In other words: 
these studies are not very helpful to the general design of systems. “Eval- 
uation of CSCW applications requires a very different approach, based on 
the methodologies of social psychology and anthropology (...) And the re- 
quired methods are generally more expensive, more time-consuming, and 
less precise” (Grudin, 1988, p.87). Others state that it is difficult or impos- 
sible to create a group in the lab that will reflect the social, motivational, 
economic, and political factors that are central to group performance 
(Malone, 1985). Group interactions or group dynamics unfold over days 
or weeks (cf. section 2.2.2). There are many statements, also as part of this 
thesis, that workplace studies are essential und more of them need to be 
conducted. Schmidt emphasizes their importance (Schmidt, 1998) even 
before writing about “The Problem with Awareness” in 2002. 

Yet, there are also different opinions. For example, Bardram (1996) 
states: 


“From the very beginning, workplace studies have played a prominent role in the re- 
search field of CSCW. They are used to understand and shed light on work and interac- 
ion happening in a workplace (...) and as such [have provided] an important insight into 
he subtleties of (...) socially constructed work practices. Within CSCW the value of these 
insights into the social nature of work activities, gained through such workplace studies, 
is unquestionable. However, there has been an ongoing dispute in the field (...) [as] to 
he exact value of these often very detailed and specific investigations of the workplace. 
Questions like: how effective is the field study approach for informing the design of 
CSCW systems? How can typical ethnographic field studies which take months or years, 


be done with the fast pace of systems development? What should be used for within the 
design process? Are they economical or even practically desirable in a complex design 
process? Is it possible to generalize such detailed and narrow studies into applicable 
design recommendations?” (p.613) 


Bardram and Hansen (2010) later add: “Most studies of plans and sit- 
uated work have applied ethnographic methods and thus fail to provide 
any quantitative insight into the extent of this phenomenon” (Bardram & 
Hansen, 2010, p.331). Further problems: 
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e Social studies yield no design guideline, no practical impact (cf. 
section1.2) revealing a gap between results and their applicability. 

e There is no transfer between studies and application design due 
to separate groups of researchers (Plowman et al., 1995). 


Later other approaches to the evaluation of groupware entered the dis- 
cussion: 


“Many researchers believe that groupware can only be evaluated by studying real collab- 
orators in their real contexts, a process that tends to be expensive and time-consuming. 
Others believe that it is more practical to evaluate groupware through usability inspec- 
tion methods. Deciding between the two is difficult, because it is unclear how they com- 
pare in a real evaluation situation” (Steves et al., 2001). 


Other researchers mention that “most discount methods rely on some 
understanding of the context in which the groupware system will be used” 
(Pinelle et al., 2003, p.281). However, others suggest the application of the 
mechanics of collaboration (Gutwin & Greenberg, 2000) for exactly these 
discount methods which make them potentially independent of the social 
context. All in all, it strongly depends on what is meant by context and 
how much you need of it. That is, the goal of the evaluation should decide 
on the method being used and not only if it is a group context or not. 


4.6.8 The “Evaluation Crisis” 


The previous section and section 3.3.4 on the cognitive versus the social 
science describe the core movements that lead to the “evaluation crisis” 
(Neale et al., 2004). “The evaluation of distributed CSCW systems has 
been too frequently method driven by various disciplinary preferences ra- 
ther than driven by frameworks that get the appropriate questions an- 
swered” (Neale et al., 2004, p.112). Further, looking at a typical develop- 
ment cycle reveals that without proper evaluation no design activity can 
be finished successfully. Research in CSCW due to its evaluation disputes 
is basically confined to its first development cycle and thus with hardly 
any progress possible. The dichotomy of ethnography or technology drove 
out cognitive science and tries to hinder research happening in rather con- 
text-independent approaches. Especially cognitive engineering seeks con- 
structs or concepts to understand human performance. It “puts forward 
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theoretically based constructs and test their generality through empirical 
studies in a wide variety of laboratory tasks, simulators, ‘microworlds’, 
and actual” (Parasuraman et al., 2008, p.140). “By empirical evidence, we 
mean controlled experimental studies and/or validated computational 
models and not just subjective observations, analytical exercises, or per- 
sonal opinions” (Parasuraman et al., 2008, p.143). The design of CSCW 
applications or groupware fails at the evaluation stage due to the lack of 
appropriate methods. In the long run, this has already caused the frag- 
mentation of the entire research field on the large scale (Kostakos, 2015; 
Liu et al., 2014; Schmidt, 2009). 

Wallace et al. (2017)’s more recent study does not indicate any impro- 
vement in this matter: 


“Our review shows that research methods at CSCW have changed with the emergence 
of social networking and Post-PC devices over the past decade, but that these changes 
have followed rather than anticipated technological advances. In particular, CSCW now 
places most emphasis on research that describes collaborative work environments in 
practice, as opposed to work that develops novel systems in the laboratory or tests scien- 
tific hypothesis” (Wallace et al., 2017, p.2). 


That is, the evaluation crisis is not yet over and will not be soon, unless: 
“The choice of evaluation methodology - if any - must arise from and be 
appropriate for the actual problem or research question under considera- 
tion” (Greenberg & Buxton, 2008, p.111). 


4.7 Summary 


The development of CSCW applications which includes their design and 
evaluation is a complex task. It was shown how CSCW researchers go 
about the risks and issues employing an evolutionary design process 
equipped with methods for the particular research context. 

This chapter’s key findings are: 
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e The CSCW design challenge is a wicked problem. Therefore, the 
development requires an evolutionary design process involving 
users already at the early stages avoiding the risk of changing re- 
quirements and the users being influenced by the resulting sys- 
tem. 

e = =©There are a lot of evaluation approaches but only a few come close 
to the requirements afforded by the characteristics of awareness 
and coordination. For instance, using a post-task questionnaire is 
likely not to capture the right values as secondary task knowledge 
is ephemeral by nature. 

e The evaluation of awareness and coordination requires this sec- 
ondary task knowledge to be still present. 

e Awareness and coordination cannot be judged in terms of effort 
by directly asking users about it or by observation as it is mostly 
a subconscious and unintentional activity resulting in ephemeral 
knowledge quickly gone when not needed. 

e Most evaluation techniques can be used for hypothesis genera- 
tion, but only very few for hypothesis validation. 

e The usage of evaluation approaches in general depends on the 
development stage (early, late). 

e CSCW currently suffers from an evaluation crisis as most evalu- 
ation methods are chosen upon disciplinary preference instead 
of appropriateness and many of them do not allow for hypothesis 
validation which is, however, required to finish a design cycle. A 
design cycle is incomplete without a proper evaluation. Thus, 
CSCW fails at the evaluation stage of the design process. 

e As coordination and awareness support mechanisms are context 
independent features it is appropriate to study these as part of 
dedicated laboratory studies as only those allow the connection of 
cause and effect. 

e Measuring efforts in the context of awareness and coordination 
requires a quantitative approach that focuses on the behavior and 
not the attitude of a user. 


However, there is no one size fits it all approach. McGrath (1995) states 
that researchers are always trying to maximize generalizability, precision, 
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and realism. Yet, this cannot usually be done in a one-step approach ap- 
plying a single method. Multiple methods must be used in combination 
with others to balance their shortcomings (Neale et al., 2004). 

When it comes to evaluations the guiding principle should be follow- 
ing: “The choice of evaluation methodology - if any - must arise from and 
be appropriate for the actual problem or research question under consid- 
eration” (Greenberg & Buxton, 2008, p.111). 


5 Standardized Coordination Task Assessments 
(SCTA) 


This section presents a new approach to the selective, early and appropri- 
ate assessment of awareness and coordination support. This chapter starts 
with the definition of the research goals and objectives connecting the ap- 
proach with a rationale. Then it takes look at the method’s concept by dis- 
cussing high-level requirements, the ideas and related work that led to the 
eventual design of the method. Next, this chapter takes a look at how 
awareness and coordination can be operationalized in terms of effort 
based on the characteristics gathered in chapter 1 and 3. The sections de- 
scribing the approach focus on the structure of the method and what com- 
ponents are used as well as its overall procedure. In the following section, 
the implementation of the approach is described including technologies 
used for its realization also drawing a little history on how the approach 
itself evolved over time. Finally, this chapter presents possible means of 
analysis based on this approach. It concludes with a short evaluation of 
the method itself with respect to its scientific reliability and validity as well 
as to its initial goals and mission statement. 


5.1 Rationale 


As mentioned numerous times throughout this thesis, one of the major 
goals of CSCW is to translate vital social mechanisms from the real world 
into technical concepts and means of support in the digital world: “CSCW 
should be conceived as an endeavour to understand the nature and re- 
quirements of cooperative work with the objective of designing computer- 
based technologies for cooperative work arrangements” (Schmidt & 
Bannon, 1992, 2013, p.351). As part of this endeavor, researchers found 
that team members coordinate their activities seamlessly and effortlessly 
in the real world and thus tried to translate this to digital system. Gross 
(2013) framed the term effortless coordination as follows: 
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“the concept of effortless coordination — that is, the question whether and how team 


work can be coordinated and a mutual understanding in the team can be gained and 


maintained, while still keeping the team members’ coordination effort to a minimum” 
(Gross, 2013, p.427). 


But how and when can coordination be considered effortless in digital 
systems? This is a tough question which cannot be answered directly but 
has to use an approximation and benchmark approach consisting of the 
following steps: 


1. 


AWN 


The coordination support is built based on a certain hypothesis 
(e.g., derived from ethnographic studies). 

The effort is measured. 

The results are compared to previous results (benchmark). 

If the effort is lower (approximation) it appears to be a hypothesis 
worth pursuing (new benchmark) otherwise it created more effort 
than previous approaches and does not help towards the goal. 


These are exactly the steps of the scientific method introduced in sec- 
tion 4.2. However, the problem is that there is currently no way to meas- 
ure or assess efforts related to coordination (and thus also related to aware- 
ness) in CSCW. 

From this situation, the following research hypothesis for this thesis 
was be derived: in order to reach effortless coordination in digital systems, 
coordination (and awareness) need to be made measurable and to be 
measured. 

This hypothesis was broken down into three objectives: 


Objective 1 - make awareness measurable: build a construct for 
awareness (conceptual operationalization). This objective will be 
met by reviewing existing literature on the subject from the area 
of CSCW gathering typical characteristics and underlying pro- 
cesses. 

Objective 2 - make coordination measurable: build a construct for 
coordination (conceptual operationalization). This objective will 
be met by reviewing additional existing literature from the area of 
CSCW, HCI, human-factors and cognitive sciences. 
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e Objective 3 - create a measuring method: design a measurement 
approach to assess awareness, coordination and respective sup- 
port systems. This objective will be met by suggesting a method 
and to demonstrate its applicability with reliable and valid results. 


Objective 1 and 2 have been addressed earlier as part of the previous 
chapters. This chapter especially focuses on objective 3: Find a way to 
measure efforts in coordination and also awareness as it is closely related 
to it. Here, the goal is to design a new method that properly regards the 
characteristics of the constructs of awareness and coordination in order to 
gain deeper insights into how the constructs can be translated successfully 
to the digital world to render them effortless here as well. 

In order to do so, the constructs need to be understood precisely, i.e., 
there is a need for the fundamental understanding of both constructs, yet, 
with a strong consideration of the eventual use already in mind. This in- 
tersection of fundamental understanding and consideration of use are the 
characteristics of use-inspired basic research, also known as research in 
Pasteur’s quadrant (Stokes, 1997) (cf. section 1.4). The above-mentioned 
goal is considered to be reached once a method is found that delivers reli- 
able and valid results in terms of effort for awareness and coordination. 


5.2 Concept 


This section undertakes the next steps from the aforementioned objec- 
tives to the concept, i.e., describing the what, of the desired measurement 
approach. In a first step, high-level requirements are gathered and derived 
from the previous objectives (cf. section 5.1). The next section takes these 
and describes ideas and major influences they are based upon. A section 
on the concept’s scope describes which of the ideas are realized to what 
extent as part of this thesis.!7 The following sections then outline the op- 
erationalization of the constructs and further thoughts on the research 
design. 


7 Out-of-scope parts can be found as part of the future work section in chapter 7. 


160 | Standardized Coordination Task Assessments (SCTA) 


5.2.1 High-Level Requirements 


The following high-level requirements for the measurement approach 
were gathered throughout this thesis and derived from the research ob- 
jectives (cf. Table 5-1). 


Table 5-1. High-level requirements. 


ID | Title Description 

1 Comparative Approach As described in section 5.1 the approach requires 
a before/after or split (i.e., A/B) procedure to allow 
approximation and benchmarking. 

2 Approximation and Bench- | Both are required in order to approximate effort- 

marking less coordination. 

3 Inexpensive Field studies are quite expensive and a quantita- 
tive approach should be inexpensive due to poten- 
tially high volumes of participants. 

4 Formative evaluation The approach must be able to be applied in set- 
tings where the solution is not yet finished. 

5 Easy replication Retries and results ofthe approach shall be easily 
replicable in order also to be reproduced by other 
researchers. 

6 Quantitative measure The approach must use a quantitative approach in 
combination with an observation and measure- 
ment ofthe users’ behavior. 

7 Distributed assessments The approach must allow for distributed and co- 
located testing. Especially distributed settings are 
an obstacle to current ethnographic studies 
(Neale, 2004). 

8 Environment integration In order to quickly integrate into existing environ- 
ments or to quickly setup its own for conduction 
experiments, the approach must not require any 
software installation on the participants device. 
All integration must be done as configuration. 

9 Device independent The approach’s implementation must not be de- 
pendent on a specific device type. 

10 | Operating system inde- | The approach’s implementation must not be de- 

pendent pendent on a specific operating system. 

11 | Distinct levels for coordina- | The approach must measure distinct levels of sec- 

tion and awareness ondary task knowledge on awareness and coordi- 
nation. 

12 | Appropriate consideration | The approach must consider awareness and coor- 

of awareness and coordina- | dination appropriately according to their charac- 
tion constructs teristics outlined in their constructs. 
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13 | Early stages of development | The approach must be applicable in the early 
stages of development of awareness and coordina- 
tion support systems. 

14 | Hypothesis validation The approach must be usable for hypothesis vali- 
dation as part of a human-centered design ap- 
proach. 

15 | Iterative, incremental appli- | The approach must allow to be applied numerous 
cation times without being affected by the effects of re- 
peated use (e.g., learning). 

16 | Re-usable/sharable results The approach must allow to (re-)use and share re- 
sults with other researchers in order to allow rep- 
lication. 

17 | Scalability The approach must allow the assessment of vari- 
ous group sizes in a configurable number of as- 
sessments. 


5.2.2 Idea and Major Influences 


The initial idea ofthe approach was conceived in 2011 (Oemig & Gross, 
2011). The idea is comprised of an easily replicable, standardized primary 
task to be solved by a team which is interrupted by a configurable number 
of freeze probes to query the participants about their secondary task 
knowledge. Based on this, the approach received its name: STANDARDIZED 
COORDINATION TASK ASSESSMENT (SCTA). 

The SCTA is a concept-oriented framework (cf. section 4.4), i.e., it de- 
fines a method to measure specific concepts (in this case awareness and 
coordination). The concepts have been specified in detail in chapter 1 
(awareness) and chapter 3 (coordination). The following subsections 
briefly describe the ideas and major influences behind the approach. 


5.2.2.1 Context Independence 


The first idea builds on the independence of awareness and coordination 
mechanics from a specific context (as shown in sections 2.4.7 and 3.3.6) 
as they are part of the mechanics of collaboration (Gutwin & Greenberg, 
2000). It thus basically allows awareness and coordination mechanics to 
be tested outside and independently from the original group or task con- 
text that there was so much discussion about (cf. section 4.6.7). This espe- 
cially allows the implementation of requirements #3 and #15 (cf. Table 
5-1). 
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The idea itself is heavily influenced by stack-based approaches from 
computer science where the lower layers are agnostic of the context of the 
higher layers as in the TCP/IP or DoD stack (Cerf & Cain, 1983) or the 
ISO OSI model (ISO, 1994) (cf. Figure 5-1). 


Application Application 


Presentation 
Utility TE 
Session 


Internetwork Global Network 


Network Network 
Physical Physical 


DoD Internet Model ISO Model 


Figure 5-1. DoD model (Cerf & Cain, 1983) and ISO OSI model (ISO, 1994). 


These stacks are the role model to build a stack using the layers of 
communication, coordination and cooperation/collaboration (cf. Figure 
5-2). A stack of underlying processes was as also suggested by coordina- 
tion theory (cf. Table 3-2 in section 3.2.1). 
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Figure 5-2. The stack underlying cooperation and collaboration. (1) (Bair, 1989)’s horizontal 


chain from inform to cooperate has to be flipped to a vertical position (2). As people either 


collaborate or cooperate based on their coordination, both are set to the same level (3). Step 


(4) then adds the different layers for the coordination layer, based on the MHP (Card et al., 


1983) and the findings from this thesis with regard to the relationship of awareness and 


coordination (cf. section 3.3.6). 


As part of the stack, the STANDARDIZED COORDINATION TASK 
ASSESSMENTS especially focus on the two mechanics of collaboration for 
awareness and coordination (cf. Figure 5-3). 


Collaborate 


Coordinate 


Awareness 


Cognitive Subsystem 


Perceptual Subsystem 


Interact 


Figure 5-3. SCTA area of application. 


5.2.2.2 Interdependency Type Dependency 


Application 


Mechanics of 
collaboration 


Context dependent 


Context 
independent 


Ethnographic studies 


The second idea to be introduced here builds on the interdependency 
types of Malone and Crowston’s (1990) coordination theory (cf. section 
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3.2.1) and the previous insight, that awareness is a support function to 
coordination and that each awareness type can be matched to a coordina- 
tion interdependency type (cf. section 3.3.6). This allows an even stronger 
focus (meaning a shift from context independent to being dependent on 
the interdependency type). That is, there are overall three generic contexts 
that awareness and coordination mechanics can be observed and evalu- 
ated in. For these three generic contexts three applicable primary tasks 
need to be defined to assess the resulting secondary task knowledge. 
These generic primary tasks are also referred to as coordination games as 
SCTA experiments use a game-like setup. From another angle these coor- 
dination games are the standard tasks the name of the method refers to. 


5.2.2.3 Cognitive Approach 


The next idea described here is to use a cognitive approach, as awareness 
and coordination are based on mental processes and models. Card et al. 
(1983) state: “An applied psychology that is theory-based, in the sense of 
articulating a mechanism underlying the observed phenomena, has ad- 
vantages of insight and integration over a purely empirical approach” 
(p.13). Additionally, cognitive approaches like Card et al.’s (1983) MHP 
(cf. section 3.1.1) are partially based on metaphors borrowed from com- 
puter science like the one that sees the human mind as information pro- 
cessor. Especially talking about workload and effort again suggests a met- 
aphor like a task manager of an operating system displaying current loads 
and capacities (cf. Figure 5-4). 

The second part of the idea regarding a cognitive approach is the pre- 
vious measurement of workload in other settings outside of CSCW and 
computer-supported cooperation. 


“One of the most widely used techniques to measure workload is the secondary task. 
This technique requires an operator to perform the primary task with that task’s speci- 
fied requirements and to use any spare attention or capacity to perform a secondary task. 
The decrease in performance of the secondary task is operationally defined as a measure 
of workload” (Gawron, 2008, p.94). 
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Figure 5-4. Task manager metaphor from computer science applied to psychological con- 


cepts of workload and capacity for a human information processor (Card et al., 1983). 


For the context of SCTA this approach is basicallyturned upside down. 
The SCTA uses an arbitrary primary task (or better one, that fits one out 
of the three interdependency types) that creates an initial workload and 
especially the need for the secondary task (i.e., awareness and 
coordination). The primary tasks can be tuned up, if needed, but in 
general they serve as the base load to make experiments comparable and 
to have equal or very similar workloads across all of them. That allows 
then to change only one variable in the experiment (the 
awareness/coordination support) and to observe the effects on the mental 
effort. 

Last but not least the setup of the first primary task was also borrowed 
from cognitive psychology, where especially memory experiments were 
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conducted using abstract syllables or letter combinations (cf. section 
3.1.3). More details on this are described in the next section. 


5.2.2.4 Subliminal Messages and Letter-counting 


The idea for the first coordination game for the interdependency type of 
simultaneity was borrowed from the context of subliminal messages 
(Karremans et al., 2006; Stern, 2015). Here, researchers also used an arbi- 
trary letter counting activity (i.e., participants were counting Bs on a 
screen while being exposed to subliminal messages). The basic idea for 
this research stems from a private market researcher named James 
Vicary, who claimed in a study to increase sales of beverages and popcorn 
by using subliminal messages in movie theatres (cf. Figure 5-5).!8 


Figure 5-5. Vicary’s idea of subliminal messages in movie theatres (Karremans et al., 2006). 


5.2.2.5 Automation and Probing 


Automation and probing actually refers to two separate ideas, yet their ef- 
fect basically comes as one. Automation is the answer to the requirements 
of being inexpensive and replicable providing similar workloads for the 
primary task when running a set of experiments. The automation creates 
the letter counting task. By using all letters of the alphabet, a large number 


18 Vicary’s study was obviously a hoax, however, the context here does not refer to his study 
but to the idea of using letter counting when trying to replicate his suggested findings 
(Karremans et al., 2006). 
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of different primary tasks can be created instantly using a random letter 
generator. But not only the task is generated automatically. The questions 
as part ofthe probing are automatically generated as well based on previ- 
ously entered results from the participants. 

The probes used are similar to the ones used in SAGAT (Endsley, 
1988, 2019; Endsley et al., 1998) (cf. section 4.5). The idea is to also blank 
the screens of the participants multiple times during the task delivering 
questions about the task. The major differences to SAGAT are, that the 
questions address coordination and awareness indirectly and that the 
questions are completely autogenerated based on letter counting results. 


5.2.2.6 Standardization 


The idea of standardization is to allow the sharing and replication of re- 
search results. 


“A standardized questionnaire is a questionnaire designed for repeated use, typically 
with a specific set of questions presented in a specified order using a specified format 
with specific rules for producing metrics based on the answers of the respondents” 
(Sauro & Lewis, 2012, p.185). 


Advantages of standardized approaches (Sauro & Lewis, 2012, pp.185) 
are: 

e Objectivity: standardized measures allow other researches to inde- 
pendently verify the results of others. 

e Replicability: replication of other and own results is easier with 
standardized procedures. 

e Quantification: standardized approaches allow researchers to re- 
port results in finer detail and to apply statistics to their results. 

e Economy: standardized approaches cause great effort in their de- 
velopment but are very economical to reuse. 

e Communication: it is easier to communicated standard measures 
as the rules of their creation are available to all. 

e Scientific generalization: standardization is essential for assessing 
the generalization of results. 


168 | Standardized Coordination Task Assessments (SCTA) 


5.2.3 Scope 


The scope of the method for the purpose of this thesis is reduced in two 
ways: 

1. Awareness/Coordination support/mechanics are independent of 
the application that is used in a particular context. Thus, they can 
be dealt with in an isolated fashion. The SCTA focusses only on 
the coordination space described as part of an Clover architecture 
(cf. (Ellis & Wainer, 1994; Laurillau & Nigay, 2002) or section 
2.4.7). 

2. The implementation of the method starts with the coordination 
game for the interdependency type simultaneity. 


In order to explain the second scope reduction of the approach it is 
best to use a mixture of the Time-Space Taxonomy (Johansen, 1988) in- 
troduced in section 1.1 and the interdependency types of coordination as 
part of the coordination theory (Crowston et al., 2006; Malone & 
Crowston, 1990, 1994) introduced in section 3.2.1. Figure 5-6 shows that 
interdependency type simultaneity predominantly occurs in settings 
where the team members work at the same time (either co-located or re- 
mote), whereas the other two types of prerequisite and shared resource 
handle the time in-between multi-user sessions. 


Same Time Different Times 
Simultaneit Prerequisite / 
same Plage y shared resource 
Different Places Simultaneity Prerequisite / 
shared resource 


Figure 5-6. Time-Space Taxonomy coordination interdependency types. 
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This reduction in scope has an impact on the results later to be applied 
to applications like the following: Google Docs!? by Google Inc. (cf. Figure 
5-7)is ashared word processor. Here, multiple users may create or co-edit 
shared documents at the same time while their presence and the activity 
is indicated by colored cursors. 
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Figure 5-7. Google Docs. Multiple users are indicated by colored cursors (Source: Google 
Inc.) 


Another example for the case of simultaneity are multiplayer games 
like Star Wars™ Battlefront II by Electronic Arts Inc.2°. Here multiple 
users may play together on a common mission with the activities and 
whereabouts of the others indicated by an awareness widget named radar 
view (cf. section 2.3.1 and Figure 5-8). 


19 https://docs.google.com 
20 https://www.ea.com 
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Figure 5-8. Multiplayer games like EA’s Battlefront use radar views (in the lower left) for 


coordination (Source: EA Inc.) 


5.2.4 Operationalization 


This section briefly describes the constructs’ operationalization for the us- 
age as part of the STANDARDIZED COORDINATION TASK ASSESSMENTS 
(STCA). The starting point is the stack-based model in Figure 5-9 which 
was introduced in section 5.2.2.1. 

The part covered by the SCTA reaches from the perceptual system to 
coordination. Starting from the bottom, the perceptual subsystem and the 
cognitive subsystem were described as part of the Model Human Proces- 
sor (MHP) in section 3.1.1. 
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Figure 5-9. The stack-based model. 


Here, the memory entities are described by the following parameters 
(Card et al., 1983, p.25): 

e The storage capacity in items (u) 

e The decay time of an item (5) 

e The main code type (physical, acoustic, visual, semantic) (x) 


Whereas the most important characteristic for a processor is its cycle 
time (t). Overall, Card et al. (1983) even published values for these param- 
eters (cf. Appendix B). However, for assessing the effort of coordination 
the values alone at this level do not suffice. As mentioned earlier, the start- 
ing point for the operationalization of awareness and coordination, was 
the single aspect of effort as in ‘effortless coordination’ (Gross, 2013). Ef- 
fort (cf. section 3.1.5) can be defined as “the conscious exertion of power 
or total work done to achieve a particular end” (Merriam-Webster, 2017). 
Basically, it expresses the amount of rigor and energy a user needs to 
spend when dealing with his secondary task knowledge which competes 
with the primary task knowledge for the limited capacity of the working 
memory. As described in section 3.1.5, effort is measured in terms of 
time. 

In the real world, articulation work (Strauss, 1985) is the effort of coor- 
dination tasks and responsibilities between distributed collaborators. This 
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type of work can be reduced by developing divisions of work, conventions 
and by the use of coordinative artifacts (Bardram & Bossen, 2005). As far 
as the stack in Figure 5-9 is concerned it reveals three areas of effort. An 
area of effort is a specific region of the construct for which the effort can 
be determined separately. Two of them, the secondary tasks of awareness 
and coordination, are in scope for the SCTA. The third one, that is the 
primary task or the cooperative/collaborative task, is not the main focus 
here but is also measured. This is due to the fact that primary and second- 
ary task knowledge share the same memory capacity (cf. section 3.3.2) and 
if the secondary task affords too much effort then this could be observed 
in the performance of the primary task. 


Areas of effort 


Working Memory 


Situation Awareness Coordination 


Perception of Comprehension of | Projection of Future 
State of the Elements in Current} Current Situation Status Problem- cs Performance 
Environment Situation solving Decision Kei 
(Level 1) (Level 2) (Level 3) cons 
Î 
Long-term memory 
+ Goals & Objectives 
+ Preconceptions (Expectations) 


Figure 5-10. Three areas of effort: awareness (secondary task), coordination (secondary task) 


and the performance of actions (primary task). 


The effort area of awareness covers the perception of elements, the 
comprehension of the current situation and the projection of future sta- 
tus. “When we are talking about ‘awareness’ we are talking about the phe- 
nomenon that actors align and integrate their activities with the activities 
of others without interrupting the current line of action and in a seem- 
ingly effortless way” (Schmidt, 2002, 2011b, p.162). 

The effort area of coordination includes problem-solving and decision- 
making efforts (cf. Figure 5-10). “Situation awareness forms the critical 
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input to — but is separate from — decision making, which is the basis for 
all subsequent actions” (Endsley et al., 1998). 

The idea of how to actually measure the effort in the above-mentioned 
areas stems from the following quote: “These practices appear effortless 
because the indicators of states or state changes in the field of work and 
of colleagues’ intentions are ready-at-hand or are easily made ready at- 
hand” (Schmidt, 2011b, p.194). 

For the effort area of awareness, this leads to the following assump- 
tions: 

1. IfI am aware about something, I can answer probe questions 

about it quickly and correctly. 

2. The longer the answer takes, the more effort is involved. 

3. The effort needed to answer the probe question is proportional to 

the regular effort during the task without a probe. 


The assumptions basically ask for how long it takes to retrieve the re- 
spective information or to make a certain decision which corresponds to 
the effort in that particular area. But there is already an additional aspect 
in these assumptions: the quality of awareness information. Besides the 
effort, the fact whether the information was stored and retrieved correctly 
soon became another aspect for the quality of awareness and coordina- 
tion. For the case of awareness, the above assumption was formulated that 
if somebody is aware of something, then they can answer questions about 
this something not only quickly (relating to effort) but also without error 
(relating to information quality). Quick and correct answers being an in- 
dicator of low effort as the information can be retrieved easily from the 
working memory. This something people need to be aware of as part of 
the assumption refers directly to the elements of awareness (cf. section 
2.3.1). However, not all of the elements are equally important at the same 
time. Their importance is determined by the interdependency type (cf. 
section 3.2.1) of the coordination required. Consequently, this accounts 
for three sets of awareness elements to focus on for the interdependency 
types prerequisite, shared resource and simultaneity. For instance, for the 
type shared resource, the user requires knowledge about its availability 
and whereabouts while for the type of simultaneity this need is replaced 
by the one for current activities. These shifts in significance of awareness 
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information requirements induced by interdependency type changes can 
be observed for example in instant-messaging applications (Oemig & 
Gross, 2007): not conversing the user relies on presence and availability 
information (interdependency type: shared resource). Once conversing 
the information shifts to a typing indicator that helps to avoid collusions 
in communication (interdependency type simultaneity). This shift re- 
verses once the conversation is concluded. 

Coordination and awareness are two different concepts. While aware- 
ness relates to the present and past to provide a basis for coordination, the 
latter is rather about the future and decision making. This has to be re- 
flected as part of the operationalization. The core assumption regarding 
coordination builds again on the aspects of effort and correctness, yet, it 
is about selecting options for future actions. That is, for the effort area of 
coordination, a different set of assumptions is used: 

1. If I know my options, I can make correct and quick probe deci- 

sions. 
The longer a decision takes, the more effort is involved. 

3. The effort needed to make the probe decisions is proportional to 

the regular effort during the task without a probe. 


Getting the above information requires asking users during a collabo- 
rative task about their secondary task knowledge. The answers should 
measure the response time and whether the answer was correct or not. 


5.3 Approach 


This section describes how the operationalization (cf. section 5.2.4) is 
transferred into practice by using some of the ideas mentioned in section 
5.2.2 while at the same time regarding the high-level requirements (cf. 
section 5.2.1). As mentioned in the scope description (cf. section 5.2.3) the 
subsequent approach only covers the interdependency type of simultane- 
ity. This section first describes the structure of the approach and then its 
procedure. 


Standardized Coordination Task Assessments (SCTA) | 175 


5.3.1 Structure 


The structure of the approach is comprised of the primary tasks, the 
probes and questions as well as the overall organization in runs, sessions, 
and traces. 


5.3.1.1 Primary Task 


In order to implement the questioning of subjects regarding their second- 
ary task knowledge requires a primary task. This task is something the 
subjects work on in a collaborative manner as the task requires simulta- 
neous cooperation. Inspired by the letter counting activity as part of the 
research on subliminal messages (cf. section 5.2.2.4), this approach uses 
a letter counting activity as well. A team of people is asked to jointly count 
all letters presented to them. This approach differs in a number of ways 
from the one used by Karremans et al. (2006): 

e Itisa shared collaborative task as opposed to a single-user task in 
the context of subliminal messages. 

e The goal of the primary task workload is to cause secondary task 
knowledge and not to divert participants from outside influences 
like subliminal messages. 

e It uses all letters of the alphabet instead of just one that is to be 
counted. 

e The set of letters is automatically created using a random genera- 
tor assuring an equivalent effort required by participants when 
running the experiment. 

e Due to the changes to the set of letters subjects may participate 
multiple times in the experiment as the presented letters are al- 
ways different as opposed to the task with subliminal messages. 


The counting activity and the required coordination creates the mental 
load to be measured in terms of the effort-based operationalization outline 
earlier (cf. section 5.2.4) 


5.3.1.2 Probes and Questions 


The counting activity is interrupted by multiple freeze probes (cf. section 
5.2.2.5). They focus on the user’s secondary tasks, i.e., why actual counting 
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results are not the most important aspect. The SCTA uses four dedicated 
question types—two concerning self-awareness and two concerning 
group awareness (cf. Table 5-2). The questions focus either on people or 
artifacts (i.e., letters) (cf. section 2.1.1). They are directed towards the past 
as awareness is the knowledge of the past that provides the basis for future 
decisions. The difference between the types SELF_WHO-GROUP_WHO 
and SELF_WHAT-GROUP_WHAT is the letter that is being asked for. 
The SELF_WHO always probes for letters a user himself has counted - 
hence questioning the users about their own activities. The 
GROUP_WHAT always questions for letters counted by another team 
member while the SELF_WHAT always asks for letters that have been 
counted by the subject being asked. 


Table 5-2. Question type overview for awareness. 


Question type Focus Time Target Example 
SelfWhoQuestion Team Past Self-awareness | Who counted As? 
(SELF_WHO) (counted by self) 
SelfWhatQuestion Artifact Past Self-awareness | Was the letter A 
(SELF_WHAT) counted? (counted 
by self) 
Group WhoQuestion Team Past Group aware- | Who counted As? 
(GROUP_WHO) ness (counted by oth- 
ers) 
GroupWhatQuestion Artifact Past Group aware- | Was the letter A 
(GROUP_WHAT) ness counted? (counted 
by others) 


For the case of coordination, it works in a similar way (cf. Table 5-3). 
Yet, there is no distinction in perspective. 


Table 5-3. Question type overview for coordination. 


Question type Focus Time Target Example 
CoordinationWhatQues- Artifact Future | Coordination May As be 
tion counted next? 
CoordinationWhoQues- Team Future | Coordination Who will count 
tion As next? 


One or multiple sets (s=-[SELF_WHO, GROUP_WHO, SELF_WHAT, 
GROUP_WHAT}) of these four questions are used to query the 
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participant—always in random order. Atthe time of writing the questions 
regarding coordination could be added as an option. No two users are 
asked the same questions, since all questions have to be personalized to 
the respective participant (at runtime) always taking into account current 
counting results. 

The questions are generated by the engine fully automated based on 
previous counting results and on an individual basis as the questions rely 
on the perspective of the individual user: If user 1 counted As, then it is 
user 1 self-awareness that s/he did. At the same time, it is user 2 group 
awareness as s/he did not count the letter. 

The questioning engine records the response times and determines 
the correctness of the answers. After answering the freeze questions, the 
team continues with the primary task until they are interrupted again by 
another freeze probe. The number of freeze probes is configurable, yet the 
number of freeze probes is typically set to at least three. 


5.3.1.3 Runs, Sessions, Traces 


The assessments are organized in runs, sessions, and traces: 

e Run: a single counting task with a configured duration where a 
version of a prototype is evaluated with a set of users. A run is 
interrupted by a configurable number of freeze probes (cf. sec- 
tion 5.3.2.2). 

e Session: a set of multiple runs using the exact same prototype. 
The results are aggregated and displayed as circle inside the 41 
diagram (cf. section 5.4.2). 

e Trace: a set of multiple sessions. While each session represents a 
certain version of a prototype, the trace depicts the evolution over 
time as each session may employ another increment. Thus, a 
trace visualizes the evolutionary path of a prototype and its corre- 
sponding hypotheses. 


5.3.2 Procedure 


The SCTA procedure consists of the stages preparation and sign up, 
counting and probing eventually followed by the aftermath (i.e., clean-up 
and analysis). 
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5.3.2.1 Preparation & Sign-up 


Before any user can start with the counting activity, the SCTA administra- 
tor has to create a run (cf. section 5.3.1.3) using the SCTA Administration 
Console (cf. Figure 5-11). 


©0©® & SCTA Administration x + 
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Figure 5-11. SCTA Administration Console at start-up. 


Once created (cf. Figure 5-12) the new run appears in the run list of 
the SCTA Administration Console. The run is still not open to participants 
as the administrator needs to open it using the button in the action col- 
umn. 
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Runs 
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Participants 
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Figure 5-12. First run created, not yet opened. 


While not open participants accessing the SCTA Participant view are 
shown the “no-run available wait screen” (cf. Figure 5-13) denoting that 
there is no open run currently available. 
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SCTA Participant nm) 


Mensch-Computer-Interaktion 
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Figure 5-13. Waiting participant when there are no open runs. 


Once the administrator opened the run, participants are able to sign- 
up for it (cf. Figure 5-14). They may choose a simple name as identifier. 
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@ SCTA Participant x mE 


A Nicht sicher | 192.168.2.103:8080/webapp/client/ x a A 


SCTA Participant 


Hey, we will start soon. Please sign up! 


[L č [sma] 
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Figure 5-14. Participant registration once a run is open. 


After their registration, the registered participants appear in the list for 
the next run in the SCTA Administrator Console (cf. Figure 5-15). 


182 | Standardized Coordination Task Assessments (SCTA) 


@ SCTA Administration x 0 


A Nicht sicher | 192.168.2.103:8080/webapp/admin/ x 8 A 


SCTA Adminstration Console 
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Runs 

Id Titel Description # of Freeze Probes Status Action 
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[Update 

Participants 

Id Name Status IP-Address 

0 Chris REGISTERED ? 

[ Update 
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Figure 5-15. List of registered participants in the SCTA Administration Console. 


Further participants may sign up to the open run until the target num- 
ber of participants is reached. In the meantime, participants are shown 
another wait screen (cf. Figure 5-16). 
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Figure 5-16. Another wait screen shown to registered participants when the opened run has 
not started, yet. 


All participants registered appear in the SCTA Administration Console 
(cf. Figure 5-17) 
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Figure 5-17. Two participants registered for the currently open run that has not started, yet. 


In order to start the run the administrator needs to push the start but- 
ton. After doing so, the run’s status changes to RUNNING (cf. Figure 
5-18) 
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Figure 5-18. Run started in the SCTA Administration Console. 


5.3.2.2 Counting and Freeze Probes 


Once the run was launched, the so-called document screen is show to all 
participants (cf. Figure 5-19). This screen has a small section atthe bottom 
where participants can enter their counting results. Depending on the 
setup ofthe run, further input options may be present, for instance, cap- 
turing the intention of the participants (e.g., “I will count Ds next.”). An- 
other part not shown in Figure 5-19 is the coordination or awareness sup- 
port system under test implementing a certain hypothesis. For instance, 
an additional window may be shown listing the counting results that other 
users typed in at the bottom of their screen. This is where the ideas of 
researchers and developers start in implementing new features or con- 
cepts as awareness or coordination support system. 
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The number of letters shown is a configuration value. It is usually 
changed depending on the size of the group working on the task. 


@@@ = @ scr Participant x + 


€ C A Nicht sicher | 192.168.2.103:8080/webapp/client/ + 62% @: 


SCTA Participant 


Please count each of the following letters in a joint effort (i.e. a=?, b=? etc.). 
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ceonr mrxyt ibhac jilyu chmuw 
jvwor boeyw uoohg puglk ofcep 
zuwav azzsy pazyr dfihl vksts 
ehcsr dzzpl Istpg ebhks hdfdy 
zfxjh dvihj byvog uixot qpapl 
Counting Result 

Letter: [a v]Quantity:| [Submit | 
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Figure 5-19. Document screen showing the counting task to the participants who joint the 


run. 


After a configurable duration, the run switches into the freeze probe 
mode, presenting personalized question to each participant (cf. Figure 
5-20). The questions (cf. section 5.3.1.2) being asked are created by the 
question engine. They query the participant about his status and 
knowledge regarding the collaborative task at hand. 
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Was the letter n counted? (GROUP_WHAT) 


Figure 5-20. Freeze probe using a GROUP-WHAT question. 


Freeze probes can also be seen from the SCTA Administrator Console 
(cf. Figure 5-21). 
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Participants 
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Figure 5-21. SCTA Administrator Console during a freeze probe. 


Further question types ofthe question set (cf. section 5.3.1.2) are asked 
during the freeze probe using a random order (cf. Figure 5-22). 
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Was the letter p counted? (SELF_WHAT) 


Yes 


Figure 5-22. Freeze probe question SELF-WHO. 


After finishing all questions of the probe, the participant is shown an- 
other wait screen (cf. Figure 5-23) that serves to synchronize activities as 
some participants might answer the questions quicker than others ready 
to continue the counting task while the remainder of the group is still busy 
answering the questions of the freeze probe. 
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Freeze probe finished. Please wait! 


Figure 5-23. Participant finished the questions and needs to wait for the other participants 


to finish as well. 


Once synchronized, participants resume the counting task using the 
document screen and the provided awareness and/or coordination sup- 
port facilities. As mentioned before, the number of freeze probes is con- 
figurable as was as the overall duration ofthe run. Once every participant 
has completed the questions from the last freeze probe, the run is over (cf. 
Figure 5-24). 
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Figure 5-24. Run completed. 


5.4 Means of Analysis 


The above operationalization allows to derive different means for analyz- 
ing the data collected by SCTA runs: 

1. Key Performance Indicators 

2. The 4I-diagram 

3. Patterns 
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5.4.1 Key Performance Indicators (KPI) 


Having operationalized awareness and coordination the way as described 
in section 5.2.4 allows to derive a number of KPIs that may be used for 
further analysis or benchmarking. 


Response-Forget Ratio (RFR) 

Average response time/forgetting time ratio is defined as the arithmetic 
mean of all response times of all runs belonging to one session in relation 
to the configured forgetting time. It is derived from the response data. 


= r_ı responsetime,. 
RFR = {fr Mm} ES 
forgetting time {rh Tn} 


Error Rate (ER) 

The Error Rate (ER) is defined as the quotient of the total number of in- 
correct answers of all runs belonging to one session divided by the total 
number of all answers of all runs belonging to the same session. It is de- 
rived from the response data. 


Rt 
7-1 incorrect, 


ER= {rn ..m} ES 


n_ (incorrect, + correct,)’ 


Coordination Error Rate (CER) 

The Coordination Error Rate (CER) is defined as the arithmetic mean of 
the number of multiple counts ofthe same letter by different users divided 
by the number of all letters counted per run. It is derived from the count 
data. 


n 
1 yo multi count, 
CER =— ) ————,,{n,--m} ES 
n Zu count, 
T= 
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Performance (P) 

The Performance (P) is defined as the arithmetic mean of the number of 
letters counted per run divided by the configured assessment run time. It 
is derived from the count data and configuration. 


Lon 
z Èr=1 count, 


== [rn ..n}Ees 
run ET btd 


5.4.2 4l-Diagram 


The data gathered from the assessment runs contain response times and 
an information about the correctness of the answer. The combination of 
these two aspects provides us four typical settings depicted in Figure 5-25. 
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Figure 5-25. The combination of speed and correctness provides four typical scenarios pro- 


vided as part of the 4I-diagram. 
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While one quadrant represents the desired state, the other three offer 
rather suboptimal cases. Each of these settings received a distinct head- 
line: 

e Ideal: in this case the users provide quick and correct answers as 
the knowledge appears to be well implemented in their mental 
model. 

e Inefficient: while still answering correctly, it takes the users a long 
time, that is, it causes them a high effort to retrieve that infor- 
mation. 

e Ineffective: the answers from the users come slow and are incor- 
rect indicating that they have been misled by something and it is 
also hard to retrieve that information from memory. 

e _Illusive: in this case the users provide quick but all wrong answers. 
The retrieval obviously causes low effort but they have been misled 
by something completely. 


Since all quadrant titles start with the letter “I” the diagram is also re- 
ferred to as the 4I-diagram. In terms of the above-mentioned assump- 
tions, an evaluation yields two of these diagrams: one diagram for aware- 
ness and another one for coordination. This later extends to even more 
when the perspective of self and others and further drill-down options are 
included. 

The 4I-diagram is used to depict traces consisting of multiple sessions 
that are aggregated from multiple runs (cf. Figure 5-26) (cf. section 
5.3.1.3). Thus, it illustrates the evolution of a prototype that represents a 
certain hypothesis. 

It is important to note that the above does not tell researchers why the 
answers come either slow or fast or why something is correct or incorrect. 
It merely reflects the current state. It is left to the researcher to judge on 
the reasons and to evaluate different or slightly adapted hypotheses re- 
garding awareness and coordination support to find out. This demon- 
strates the use of the SCTA as a comparative approach. 
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Figure 5-26. Depiction of a trace in a 4I-diagram. 


5.4.3 Patterns 


The 4I diagram usually depicts measurement results for self and group 
awareness in one aggregated value. However, as indicated in section 
5.3.1.2, response time and correct or wrong answers of questions regard- 
ing the self-perspective (SELF_WHAT, SELF_WHO) can be used to cal- 
culate the x- and y-coordinates for the average self-awareness circle in the 
diagram. The same can be done with the answers of questions for the case 
of group awareness (GROUP_WHAT, GROUP_WHO). Additionally, the 
performance indication (circle radius) can be applied to the two circles 
respectively, yielding a typical set of patterns (cf. Figure 5-27). 
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Figure 5-27. Patterns of positions and proportions depicting levels of self and group aware- 


ness. 


Due to the detailed questioning concept the radius of the self-aware- 
ness circle shows the average individual performance. The radius of the 
group awareness circle depicts the average group performance. As a basic 
validation check the group’s performance has always to be greater than or 
equal to the individual’s performance (in this case with the maximum co- 
ordination error, i.e., all letters were counted by all participants). Another 
expected behavior is that the farer apart the circles get from one another, 
the more the radiuses of both are likely to assimilate. In other words: the 
greater the quality gap of self and group awareness becomes, the more the 
group’s performance deteriorates—due to coordination problems. Over- 
all, both circles together allow observing patterns of positions (circle cen- 
ters) and proportions (circle sizes) (cf. Figure 5-27). Pattern a) shows no 
displacement but equal radiuses (eclipse) indicating severe coordination 
problems since individual and team performance are nearly the same, 
knowledge of own and other activities has the same quality; b) shows a 
displacement with the same circle size indicating severe coordination 
problems since individual and team performance are nearly the same. The 
knowledge of own activities is better than about group activities. Pattern 
c) is nearly the same as b) but with knowledge of group activities being 
better than about own activities. d) shows no displacement (same quality 
of knowledge about group and individual activities) and group perfor- 
mance is greater than individual performance indicating very few or no 
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coordination problems. e) contains a displacement (knowledge quality 
about individual activities better than about group activities) and group 
performance greater than individual performance indicating minor coor- 
dination problems. Eventually, f) shows a displacement (knowledge qual- 
ity about group activities better than about individual activities) and the 
group performance greater than individual performance indicating minor 
coordination problems. 


5.5 Implementation 


The following section share some deeper insights into the implementa- 
tion ofthe approach which originally started out as a Java fat-client which 
was later migrated into a web application. 


5.5.1 Java Fat-Client 


The SCTA Tracer (Oemig & Gross, 2012) (cf. Figure 5-28), a combination 
ofadministration console and client for participants, was developed using 
the Java programming language.?! Its overall architecture works accord- 
ing to the mediator pattern (Gamma et al., 1994). This pattern belongs to 
the object-based behavioral patterns. In this pattern a central controlling 
instance, the mediator, promotes loose coupling by keeping the collabo- 
rating objects (called colleagues) from referring to each other directly. The 
mediator controls and coordinates interaction and represents the soft- 
ware’s overall behavior. Colleagues obtain a reference of the mediator 
from a central registry. The mediator is responsible for sending/receiving 
information to/from the respective colleagues. 


21 https://www.java.com 
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Figure 5-28. SCTA-Tracer (Java implementation). 


The first to login automatically becomes the administrator of the ex- 
periment. The subsequent logins will be assigned the participant role. All 
participants are show in the list of the administrator screen (cf. Figure 


5-29). Once a sufficient number of participants is present, the administra- 
tor launches the run. 
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Figure 5-29. SCTA Tracer administrator screen. 


Once launched participants get to see the document screen (cf. Figure 
5-30) where they start with the counting activity. Also, in this implemen- 
tation the counting is interrupted by freeze probes asking the already 
known questions from a question set. At this point of the implementation 
the question set only contained questions regarding awareness (cf. Table 
5-2). Response times and correctness are recorded once a user answers 
the probe questions. 
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SBoZd LIKHp sPalt Tgsjb RoFyX hL£Tg NsEwM NiaNR iiXUD yqixu 
ATFKC bqPTR RiPIJ jFrQB AT£FG qfznB zfwfU rqXXO nGF£N xIhEZ 
esHEf TmlXF taoVa utenS WTwök UBNeQ ZmXNE rUvqR kvEmy bPQuL 
QTYsV XkULc ORPie wKgsU SfwNd djhWT OOlEh Eddeq GGhDr YnoOPw 
ODuNn yURRt XyDgP BYAye ENXQS pOOLE ygYWu KnEFO UCjDZ OSCst 
GfcGR IKzHg DDeiL HHKkb jzpCf aGamt LaogD fgONP XySxkG ujeGy 
HEysY AKS£EN WgImi qyhlR asqhY zdxOP XLSCf ATmun zUtdz aZMBu 
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Figure 5-30. Document screen show the counting task. 
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Architecture 


5.5.1.1 
As described in the introduction, the architecture of this implementation 


follows the mediator patter (cf. Figure 5-31). 
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Figure 5-31. UML class diagram using a mediator pattern. 
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There are two interfaces named ISCTATracerMediator and IS- 
CTATracerColleague. Java’s Remote Method Invocation (RMI) was 
used to allow the distributed use of the software, the two interfaces are 
Java Remote Interfaces at the same time. 

At start-up the first main () method creates an ISCTATracerMedi- 
ator instance of the remote object implementation (the stub) and tries 
to bind that instance to the name SCTATracerMediator ina Java RMI 
registry. When registered successfully, this instance launches the admin- 
istration screen (cf. Figure 5-29). Further instances try to do the same, but 
their registration as SCTATracerMediator will fail, due to an already 
registered instance. Therefore, these further instances will create objects 
of the ISCTATracerColleague interface. These colleague instances 
obtain a mediator reference using a RMI registry lookup. Now they are 
able to use the mediator’s register () method to place their remote 
interface references there. Thus, the setup of the mediator pattern is com- 
plete. In the following the colleague instances setup the wait screen for 
their users. The mediator controls the colleagues using the ISCTA- 
TracerColleague interface — for instance, when the assessment run 
starts, questions of the freeze probes are to be shown, or when the assess- 
ment run is over. The colleagues use the ISCTATracerMediator in- 
terface to (un-)register for assessment runs, to send counting results, and 
to answer freeze probe questions. 


5.5.1.2 Persistence 


For the initial version of persistence, a lightweight approach know as XML 
data binding was selected. This allowed accessing XML data using objects 
rather than using DOM or SAX. The Java Architecture for XML Binding 
(JAXB) allows mapping Java classes to XML representations. It provides 
two main features: the ability to marshal Java objects into XML and the 
inverse, that is to unmarshal XML back into Java objects. JAXB allows 
storing and retrieving data in memory in any XML format, without the 
need to implement a specific set of XML loading and saving operations. 
JAXB is part of the Java SE platform.?? 


22 https://www.java.com 
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The storage format was defined as XML Schema (cf. Figure 5-32). The 
plan was to use one XML file per assessment trace. Therefore, it became 
out top-level element in the schema. It only has a name attribute. A trace 
element may contain multiple session elements, which also have a name 
attribute. The session element may contain multiple run elements. A run 
element holds the information about the participating users, counting in- 
formation and response data from the freeze probes. 

The binding compiler xj c is used to generate a set of Java classes that 
represent the schema. These classes are filled by the application with the 
data collected. When the administrator chooses to save the current status 
of the trace then this data structure with its top-element class Trace is 
handed over to the Marshaller object to create the XML file. On the other 
hand, at application start-up an existing XML file can be chosen from 
which the Unmarshaller object creates a data structure to be used by the 
application. 


5.5.1.3 Charts 


Pushing the button “Run Evaluation” on the evaluation tab (cf. Figure 
5-28) creates a 4I-diagram using an extended version of JFreeChart??. It is 
an open-source Java framework allowing the creation of complex charts of 
various types like XY charts (line, spline, and scatter), pie charts, Gantt 
charts, and bar charts (horizontal, vertical, stacked and independent). Be- 
sides the creation of charts, JFreeChart allows the placement of various 
markers inside the resulting diagrams. However, in the case of the 4I- 
diagram custom chart was needed. Fortunately, JFreeChart proved to be 
easily extensible for this situation also due to the availability of its source 
code. Error rate and response time/forgetting time ratio are used as stand- 
ard x- and y-coordinates. However, the need to influence an item’s diam- 
eter and color by the values of performance and coordination error rate 
required customization. Additionally, an individual label was needed for 
each item of a series to be shown while standard JFreeChart allows only 


23 https://www.jfree.org/jfreechart/ 
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Figure 5-32. Trace structure (XML Schema). 
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labels per series (which corresponds to an assessment trace; a series of 
items corresponds to an assessment run). First, a new dataset type was 
defined that is able to contain all four measures per item (i.e., a 4-tuple, 
or quadruple) in addition to a session label. This dataset, named SCTAD- 
ataSet, extends JFreeChart’s XYDataset. In order to deliver the data 
on the screen a custom renderer needed to be defined that displays the 
dataset’s content as 4l-diagram. The SCTARenderer extends 
JFreeChart’s XYLineAndShapeRenderer class to do the job. The gen- 
eration of proper labels required a customized SCTAItemLabelGener 
ator. The JFreeChart object is finally added to a standard Java Swing 
container. 


5.5.2 Web Application 


The Java fat-client implementation had a couple of major disadvantages. 


€ G © localhost:8080/webapp/client, 


SCTA Participant 


Please count each of the following letters in a joint effort (i.e. a=?, b=? etc.). 


acgoj magng ifihk iwzfz xyzvr 
ouudg sqfyg szjll tubum timva 
wvcxv aycdn esotg xjevi delbb 
ekgud tfntv yqcuc xeilp yerax 
pdmgc gstcq xchau fdnjr aiomx 


Counting Result 
Letter: [a v | Quantity: Submit 


Figure 5-33. REST-based implementation showing the DocumentScreen. 


It was hard to install at a site to conduct experiments as it required the 
installation of software (Java runtime, SCTA Tracer etc.). Another prob- 
lem was that it only could be run on PCs or comparable IT devices. For 
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instance, it was not possible to run the SCTA Tracer on a smartphone. 
That was also the main reason why its front-end implementation was 
eventually replaced by a web-based application (cf. Figure 5-33). Overall, 
this made the setup at an external site extremely easy. Typically, a 
smartphone was used to setup a mobile hotspot. The Internet connection 
was not needed but every device connecting to the hotspot thus became 
part of a spontaneous shared WLAN. Participants could connect with their 
own devices or with locally provided ones. They only required a current 
browser on their machine or device (cf. Figure 5-34). 


Figure 5-34. Web-based application running on different device types. 
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The administration console was usually run from a tablet as it was eas- 
ier with it to move around and maybe assist with connection problems. 
Thus, the entire issues with software installations, configurations and fire- 
wall connections could be avoided. The only problem were older PCs that 
did not have a Wi-Fi connector. 


5.5.2.1 Architecture 


The architecture reused many former components, especially the ques- 
tioning engine. These components were wrapped by an API layer which 
provided the access to the core components via RESTful webservices. The 
front-end was replaced completely with a single-page application (SPA) 
using a new JavaScript library based on JQuery?*. The former mediator 
pattern was completely removed. 


5.5.2.2 Persistence 


The persistence layer described in section 5.5.1.2 remained unchanged. 


5.5.2.3 Charts 


The JFreeChart library cannot be used inside a browser. Therefore, it was 
replaced by GoogleCharts?? which seamlessly integrates with JavaScript. 


5.6 Solution Evaluation 


According to (Oulasvirta & Hornbaek, 2016) the outcome of research (i.e., 
the solution) can be evaluated in terms of its contribution to problem-solv- 
ing capacity which is defined by following five criteria: 

1. Significance: means that an approach addresses a problem that is 
important to the stakeholders of the research. The significance of 
the presented approach was derived from literature as shown in 
the early chapters of thesis and by publishing papers to peer re- 
viewed conferences (cf. section References). 


24 https://jquery.com 
25 https://developers.google.com/chart 
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2. Effectiveness: refers to the solution being able to solve the stated 
problem. This foreshadows the closing chapter of this thesis, but 
the presented approach found a way to measure efforts related to 
coordination and awareness thus providing a way to maybe reach 
effortless coordination one day. 

3. Efficiency: refers to the cost of applying the solution compared to 
the gain. This is a part where the SCTA especially stands out as it 
uses a very low-cost procedure that is highly scalable and which 
provides decent results (as to be demonstrated in the next chap- 
ter). 

4. Transfer: refers to the fact how well the solution transfers to other 
problems of the same kind. This is also a capacity the approach 
developed in this thesis is very good at (as to be demonstrated in 
the next chapter). The SCTA can be applied in various scenarios 
and settings where dealing with a standard primary task to learn 
something about the secondary (e.g., effects of disruption, forget- 
ting, switching contexts or dynamic awareness support). With fur- 
ther coordination games for the interdependency types of shared 
resource and prerequisite, the approach may be applied to even 
more problems. 

5. Confidence: deals with the concerns about a method’s reliability 
and validity. Reliability, i.e., the consistency of measurement can 
be proven in several ways, for instance, by test-retest or split-half 
reliability (Sauro & Lewis, 2012). This was done and recorded ba- 
sically on every occasion as a warm-up activity using the test-retest 
method. Once more results become available, the reliability 
method will be switched to Cronbach’s alpha coefficient 
(Nunnally, 1978). On the other end, the validity, i.e., the measure- 
ment of the intended attribute can easily be observed with the 
numbers for the effort going up when the counting task is made 
more complex, that is, by introducing larger syllables which are 
harder to remember. This correlation can be reproduced easily, 
yet, it remains to be mathematically proven. 
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5.7 Summary 


This chapter introduced the Standardized Coordination Task Assessment 
(SCTA). It described its rationale, the concept starting with the high-level 
requirements and the ideas that influenced its creation. The transfer of 
the awareness and coordination constructs was described as part of the 
operationalization which uses simple questions to query a subject about 
his/her knowledge on the primary task at hand. The overall procedure was 
outlined followed by an overview of the means of analysis. Further, this 
chapter provided some insights of the implementation of this approach 
before eventually closing with a brief evaluation of the solution. 

The SCTA is a simple and powerful tool that opens the door to many 
research issues related to CSCW’s overall goal of effortless coordination. 
It is the first approach that truly allows to measure these and to put them 
in perspective with other measurements, thus allowing to approximate the 
most minimal effort possible. 


6 Application, Experiments and Extensions 


This chapter presents a number of sample experiments to demonstrate 
the capabilities ofthe approach. 


6.1 


No Support 


This is a special type of test as it does not provide any type of awareness 
or coordination support. It uses a co-located setup (cf. Figure 6-1). Partic- 
ipants are simply explained the task and may talk to one another on how 
to proceed with the task in which they eventually engage in. They receive 
a standard counting task with three freeze probes. 

There are many reasons to use this kind of procedure: 


1. 


At the beginning it was a test for the overall procedure. It could be 
observed how well participants get along with the provided tools 
and the provided task. Thus, little misconceptions could be fixed 
very easily. Another important aspect was the sizing of the task 
itself for a certain number of people and finding an appropriate 
duration for the task. 

Observations: in this scenario the actual engagement in the task 
was not in focus but how people actually communicate and nego- 
tiate (also in terms of group dynamics, cf. section 2.2.2). Of special 
interests were also the strategies participants used as a common 
ground to solve their tasks. These observations are helpful for de- 
signing initial awareness and coordination support mechanics. 
That is, this procedure can be used for hypothesis generation as 
well. 

Calibration and reliability testing: this procedure was also used to 
discovery how the approach behaves in terms of reliability (con- 
sistency of measurement) and which configurations are the most 
appropriate for testing. 
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Figure 6-1. Simple assessment setup at the Mensch und Computer 2017 conference in Re- 


gensburg (Oemig & Gross, 2017) (Source: Christoph Oemig). 


6.2 Forgetting of Secondary Task Knowledge 


Another example of the approach’s application was the experiment on for- 
getting regarding secondary task knowledge (Oemig & Gross, 2016). The 
difference between the forgetting of primary and secondary task 
knowledge is an issue that has not been researched at all, as appropriate 
measurement approaches were missing. 

The human memory and its subsystems have had a strong influence 
on Human-Computer Interaction (HCI) research and resulting systems. 
This was especially shown with the sections on cognition where the Model 
Human Processor (MHP) was discussed (cf. section 3.1.1). As also shown, 
the working memory plays an important role as it is limited by its capacity 
(cf. section 3.3.2) which is even shared among primary and secondary 
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task. On the other end, forgetting was shown to be a natural mechanism 
to face this limitation (cf. section 3.1.3 ). In other computing contexts, for- 
getting algorithms were designed for a wide range of purposes, for in- 
stance, they help filtering instant messages (Seifert et al., 2007) or facili- 
tate the calculation of interest curves as part of recommender systems 
(Chen et al., 2014). Forgetting even became a core feature of the currently 
popular social app SNAPCHAT?® (Bayer et al., 2016). Here, it contributes 
the major incentive to conceal personal information. 

Forgetting curves (cf. section 3.1.3) illustrate the memory’s retention 
over time. Yet, collaborative settings have never been analyzed using for- 
getting curves. These settings are known to introduce the secondary task 
of coordination right next to the primary task itself. Consequently, the ca- 
pacity of the working memory is not only limited but has to be shared 
among primary and secondary task knowledge. However, rehearsing, as 
Ebbinghaus (1885) suggests, is not an option here. Additionally, interrup- 
tions and interferences have a devastating impact on its recall (cf. section 
3.1.3). The general idea in this experiment was to gain a solid understand- 
ing of forgetting in collaborative settings. These insights may help with 
the design of collaborative systems and coordination support systems. 


6.2.1 Approach 


Ebbinghaus (1885) himself never considered secondary task knowledge as 
he was not dealing with cooperative systems. Creating secondary task 
knowledge requires a collaborative task. For primary and secondary task 
knowledge the approach needs to measure how much of each knowledge 
type is lost over time. The result will be depicted as forgetting curve (one 
for each type of task knowledge). To gain even deeper insights the second- 
ary task knowledge can be subdivided into knowledge about a user’s own 
activities and the knowledge about the activities of others (cf. section 5.4). 
The SCTA helped to provide the primary task for this experiment. A group 
of participants has to count individual letters inside a shared document. 
During the task the group has to coordinate its counting efforts. The par- 
ticipants are also required to share their results with the group. These are 
recorded using the letter, timestamp, and participant name. To determine 


26 https://snapchat.com 
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the status of forgetting the counting task was suspended to query each 
participant. As in the original SCTA approach, the questions are based on 
the recorded counting results. Here, the participants were asked to com- 
plete a list of letters stating how many of a certain letter were counted 
(primary task knowledge) and by whom (secondary task knowledge). To 
make it more difficult the list also contained letters that were not counted 
atall (secondary task knowledge). Since the results also recorded who orig- 
inally counted the letters, the list of answers returned by the participants 
could be analyzed in terms of self and group knowledge. The regular 
freeze probe questions were not answered by the participants as the num- 
ber of freeze probes was set to zero and an external timer was used to 
trigger the paper-based probes. Yet, in contrast to the original SCTA the 
response times were not recorded as the focus was not on effort but on 
correctness after a certain period oftime. Another difference were also the 
questions regarding the primary task knowledge. This was done to be able 
to compare the forgetting curves for primary and secondary task 
knowledge. The forgetting curve diagram itself shows the percentage of 
correct answers on the y-axis and the time elapsed since the start of the 
task on the x-axis. 


6.2.2 Smoke Test & Findings 


For a smoke test on the entire procedure, a mixed setup was chosen using 
the SCTA for the counting task and collecting the results whereas the 


questioning was done on paper setup. 
> —— a 


Figure 6-2. Filled questionnaire. 
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The reason also being to find a proper approach to be implemented at a 
later point in time. Four participants (age 28 to 45, 50% male, 50% female) 
were invited to engage the counting task in pairs. They were allowed to 
coordinate and to take notes. They were also expected to share their re- 
sults. The times the experiment was suspended were a) in the middle of 
the task, b) directly after finishing the task, c) 30 minutes afterwards, and 
d) 2 hours afterwards (cf. Figure 6-3). 


#of 10 
letters 


a) b) 0) d) time 


Correct 100 


(%) 


50 


End of task (EoT) EoT + 30 min EoT + 2fime 


Figure 6-3. Forgetting curves for primary and secondary task knowledge. 


At point a) four letters were counted. The recall of primary and sec- 
ondary knowledge was the highest here. The recall of secondary task 
knowledge worked better than that of primary task knowledge. At point b) 
ten different letters were counted in total. While the recall of secondary 
task knowledge still worked well, there was a significant drop for the pri- 
mary task knowledge. Reaching point c) the primary task knowledge was 
nearly gone and also the secondary task knowledge was at lower levels. At 
point d) both knowledge types were nearly gone. 

A short interview following the test revealed that most of the partici- 
pants agreed that they still knew who counted the letter but they forgot 
how many of them. That is, the secondary task was still there while pri- 
mary task knowledge was gone. They stated that they told their partner the 
counting result, wrote it down and forgot about it. They had no further 
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use for this information while the secondary task knowledge still helped 
them to coordinate. Another statement made by one participant regarding 
the secondary task knowledge was that “I knew I did not count that letter, 
therefore it had to be him”. This basically explains why the levels of self 
and group knowledge have been nearly equally high throughout the test. 
A larger group of participants will probably eradicate the opportunity of 
this kind of conclusion. Overall, taking notes obviously relieved the work- 
ing memory from primary task knowledge leaving more room for the sec- 
ondary. The reason for high values for the primary task knowledge at point 
a) is likely to be sufficient capacity ofthe working memory (in accordance 
with Miller’s 7+/- 2 chunks). 


6.2.3 Discussion 


This small experiment showed, that the secondary task knowledge stayed 
longer in the working memory than the primary task knowledge, since it 
was continuously useful for coordination while maintaining its small 
memory footprint. Yet, the experiment also revealed room for improve- 
ments: the group size for the counting task needs to be increased at least 
by one so that the knowledge about the group cannot be directly inferred 
from the knowledge of one self. Another issue not covered at this point 
were interruption and/or interference. As secondary task knowledge is not 
stored intentionally it is expected to be highly vulnerable to interruptions 
(cf. section 3.1.3). This also the basic idea for the experiment described in 
section 6.4. Overall, this experiment showed that being able to measure 
the impacts on secondary task knowledge sets the ground for explicitly 
answering the questions on how to rehearse or cue secondary task 
knowledge to let computer-supported cooperative work be more success- 
ful and efficient. 


6.3 Going to Extremes 


This experiment rather describes a classic application of the SCTA which 
was also done partially due to calibration purposes, that is, to learn about 
the approach’s behavior under extreme conditions. Doing so the aware- 
ness-/coordination-support system paradox was found (Oemig & Gross, 
2014). An experiment was designed contrasting a very simplistic 
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awareness support strategy and an automated coordination support strat- 
egy. Additionally, a control group participated with no support at all. 


6.3.1 Setup 


Three types of settings were defined for measuring. The first setting in- 
troduced a simple awareness support system. It implements a simple 
awareness strategy that notifies the user about what letter another user is 
about to count (i.e., their intention) and about actual counting results 
(shown as “awareness display” in Figure 6-4). 

The second setting utilizes an automated coordination support strat- 
egy, that was learned from the “no support” experiments (cf. section 6.1): 
the total number of letters was simply divided by the number of partici- 
pants. Each user was assigned a respective fraction thereof. Once a letter 
is counted it was dashed in the coordination support window (shown as 
“coordination display” in Figure 6-4). Thus, the user does not need to 
know what the others are counting. It is the implementation of an explicit 
coordination support system (cf. section 3.2.3) that actually coordinates 
the counting effort. As they have been reported as too rigid (cf. 3.3.5) at 
least some type of reaction was expected later to be found in the post-task 
interview. Yet, in terms of effort it is expected to be at a minimum in this 
case. 

A third setting, the control group received no treatment at all. In order 
to avoid or minimize cofounding influences the experiment setting is se- 
lected by random for a single experiment run (i.e., the counting task). In 
the next step, the participants are assigned to the run randomly as well. 
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Coordination Display 


aAbBc-dDeEfFgGHhilJjkKILM 


| Counting task 
I count--> | a eo 

uyGiL 

USVGS dQkMF DJupg hKyB) Qotac dOQug WIDYb UJSyK JoEKJ XtgGF 

dtuS) bxVHI FznnF qsfxE zmvdD ucDKk jXGgo UHVnA aoTBg vbOoq 


Awareness Display 
TOM: C-->19(01.04.14 20:49) 
Chris: w-->22(01.04.14 20:49) 


EKBH jcrZj tsHCg kzZii DRitF SBmSG KdKNI SDwfL wXbTp IBCTM 

spga IRnCr UJvYc pxUfR jJnYf fZqwT TXRXI gsjvp hrGeu UnkAD 
koXdc wyPtg qgBMd thpUg CFEQO ghCHe MNOTc eyjMj UOwPU hgDVT 
‚gXvJl KEZEB FeVkW TIGsr MNbYE VdEmQ QSaKW FbOPi psSfn KxvfU 
jXJcS UHLOB VGCWO uLulg bjFwC kjEAg eSjEY avUW) XSHRa CYoOC 
gTWgQ mRHOP xHuXP dsUMD gWkbD VByjT LLjLT PjITI ddyUP rFWaP 

_Khycr ieNcX WGEkM bGpPC WzkdS IfDWe vnZSU XzmQj Qwbmv upOtO 

OcyRc pdNOM nsUyy hEmQv fsEJs esXMs ZdZri dsOBc mSXOv OuXws 
UrsbY Qmjxc BWKCG XHcLQ tLyKg ujTmo rRddC ccJXi wDFQF HowWR 


Chris: w-->Start counting(01.04.14 20:49) cFkIO tVwlv GULYb tcyNc jirbl EPiEs EDAvh eakTw yDzMP of BDG 
TOM: C-->Start counting(01.04.14 20:49) XqYqq yfdAn WkPlp PIIjR TTIZs QZiwx GpuAY KdQJv NdMdc zJSNV 
HCbxC Retco inBKg IfkWa alvJl FENkM Wihmo ivco) yEJmd vGvPD 
hdMIE iXaAK IRvge ZaMtY Lr)bu UgbAT sWeyR bAahv aubDi hKcjY 
leQiGX dBfxw TQKFD kgoOf wAPdG WWDGS SLIxW drTxi PWWFg EKwMF 
SEKI EFtjv UXqKz yBuBS VrYbc SGcyg VFbyl mjzLL xQMJh sWFwW 
ZLeT GElkn DfljJb VrtIR dzRXR xqcrm HpBqc FhHog GoBui hjYUm 
gXZZY QdSxl tlwAv ycXEi xpyk) bXEtb Izbqb BwREo YUjEh vsism 
FGIBU gCkYd cctHR LCHRo TKmjU wYTgu qyHOB FcyvW vMWRI DYiVG 
VTCC sxQon JryLz IGczT FQueB ueUmM upgha WNIYa ArRXR xbJun 
cLQHn cUDNz NsWKE QRs2Z fTBEz lJutX ZnwmU FePDN Sghkd Ohr)k 
pCxiJ hPSAC dzQRI izGOh PypEl JPjaw tUlaR uEBMp yDJKN BDHIY 
fujxu wCDZc YUewF PGkxK PHpaY Zza0o ozbFJ JpKDN Catcx PpIHO 


Counting result--> 


Figure 6-4. SCTA counting task (old Java client) with awareness support and coordination 
support show at the same time. Only one of the left-hand windows was shown to partici- 


pants, depending on the group they were in. 


There was a total number of 24 participants, 34% female and 66% male 
in a range of 17 to 40 years with the majority drawn from a computer 
science class (including the teacher) of a German High School. The focus 
was on dyads, i.e., from the total number of participants groups of two 
were arranged. They used individual computers separated by blinds so 
they could not see each other. This time they were not allowed to talk or 
to communicate by other means outside the provided tools. Since multiple 
counting tasks took place concurrently, participants did not even know 
with whom they were actually working (no real names were used). A 
counting task lasted 10 minutes interrupted by three freeze probes. These 
asked the participant two questions concerning group awareness, two 
questions concerning self-awareness and two questions regarding 
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coordination (cf. section 5.3.1.2). Thus, a total of 18 questions had to be 
answered by a participant during a run. Besides the performance (total 
number of counted letters) the coordination error rate (duplicate letters) 
was measured and recorded. 


6.3.2 Results 


This section briefly describes some of the most important results of this 
type of experiment. These are derived from the response time diagram (cf. 
Figure 6-5) and the success rates, performance and coordination errors 
shown in Table 6-1. 


Unsupported 
i u Awareness Support 
E Coordination Support 


Figure 6-5. Mean response times per question area for each experiment type including stand- 


7000 


ard error. 


The experiment without any support type was the winner in the self- 
awareness category (4s, 95%) the only support participants had in this 
group was their knowledge about their own activities. Not surprisingly the 
awareness support experiment type was the most successful and fastest 
(4s, 75%) in the category of group awareness. Last but not least the coor- 
dination support type experiment performed best (4.2s, 95%) in the 
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coordination category. But the most interesting lies beyond the expected, 
which rather proves that the approach generally appears to work. 


Table 6-1. Success rates (percentage of correct answers), performance (total number of let- 


ters counted), and coordination errors (number of letters counted multiple times). 


Performance | Coordination 
Errors (Avg) 
Unsupported 1 
Awareness 2 y 0 
Support 
Coordination 5%, Yo, a. 0 
Support 


However, in the coordination category the awareness support group 
was not the fastest (5.2s) and created some errors (success rate 70%), 
which might be due to its indirect nature in supporting coordination as 
mentioned in the concept. Only the direct coordination support per- 
formed better (4.2s, 95%), while the results for coordination of the unsup- 
ported type suggest that the participants were simply guessing (4.5s, 25%). 

Furthermore, the data suggests that the coordination support does not 
require the user to have superior levels of self- and group awareness. Par- 
ticipants answered slower and increasingly wrong but overall performed 
best indicating that they do not really need this specific types of 
knowledge. This effect was named the awareness-/coordination-support 
system paradox: while awareness support facilitates coordination support 
the latter has not necessarily a need for the former. As expected, the third 
setting (no support) shows the highest number of coordination errors. 
Yet, all types showed nearly the same overall performance. This version 
of the SCTA used a large number of different countable letters (whole al- 
phabet, small and capital letters) and a large quantity of each single letter. 
Therefore, also the coordination errors were low requiring the reduction 
of the number of different letters in future experiments to provoke a larger 
number of coordination errors. 


6.3.3 Discussion 


The experiment revealed some new insights and new problems regarding 
the goal of effortless coordination: 


The 
proach: 
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The awareness-/coordination-support system paradox: this effect 
described above has the consequence that when developing a co- 
ordination support system, it needs to be carefully reviewed 
which awareness information is still needed by the user. This 
adds another possibility to create more efficiency but this has to 
be done very carefully. 

Effortless coordination’s anti-social tendency in digital systems: 
driving efficiency to the extreme may sacrifice social subtleties 
for its own purpose. Though performing best some participants 
mentioned that they did not like the explicit coordination support 
experiment type. They could not see what their partner was doing 
or if s/he needed assistance. Reducing efforts to the minimum 
also contradicts general design principles of human-work. The 
consequence is not only to measure response times and errors 
but also include means to measure satisfaction (cf. section 7.2.2). 
This is also mentioned as part of the mechanics of collaboration 
(cf. section 4.5). 

Self-awareness obviously needs no support: the unsupported ex- 
periment type actually removed all efforts concerning perception 
and comprehension from the user without offering a substitute. 
However, according to the measurement, participants performed 
even better in the category of self-awareness. As a consequence, 
self-awareness does not need to be supported releasing screen es- 
tate and other resources for other information. Yet, it needs to be 
verified if this remains true for more complex settings. 


experiment further created insights regarding the SCTA ap- 
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e The twofold letter problem: there were too many letters in too high 
quantities making it easy for the participants to avoid coordination 
errors. Further, this fact reduced the total number of letters 
counted since the participants spent longer times counting a sin- 
gle letter. This in turn reduced the effort needed to coordinate the 
situation. As a consequence, future experiments have to dramati- 
cally decrease the number of letters and their quantities. 

e Don’t know/don’t care answer options: Another point mentioned 
by participants was that for some questions they would prefer an 
“I don’t know” answer option instead of being forced to some 
other answer. This can also be derived from the results: the “I 
don’t know” answers are those where there is a quick wrong or 
deliberate answer while producing an overall high performance. 
However, it needs to be discussed if an “I don’t know” answer is 
sufficient or if an “I don’t care” needs to be introduced as well 
since some of the knowledge is really not needed as the found par- 
adox implies. 

e Freeze probes killing counting results: Another aspect mentioned 
by participants is that the freeze probes prevented them some- 
times from finishing their counting. Since there were only a few 
letters counted but those at high quantities this might have a large 
impact which can be avoided by smaller quantities for each letter 
and again reducing the number of letters overall. 


6.4 Effects of Disruption on Secondary Task Knowledge 


This section reports on another experiment (Oemig & Gross, 2019) which 
especially aimed to expose secondary task knowledge to disruption to 
study the effects. Secondary task knowledge is stored subconsciously and 
due to its ephemeral nature, it is easily compromised by interruption and 
interference which cause information to be lost (cf. section 3.1.3). Related 
work suggests a strong effect of interruptions on a collaborative task due 
to the impact on the secondary task knowledge located in the working 
memory. Yet, the major effort of scientific work focused on minimizing 
the disruptiveness (Cutrell et al., 2001), i.e., finding the right time and 
manner for interruptions, or understanding the role of context on the cost 
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ofinterruption (Mark et al., 2008). However, from a SCTA perspective in- 
terruptions can hardly be avoided or optimized entirely. The approach ra- 
ther seeks to support the finding of new ways helping users to recover the 
best way possible from various types of interruptions during their collab- 
orative task using appropriate support mechanisms and cues. As there are 
different types of interruptions (some of which we introduce later) this at 
least calls for adaptive coordination or awareness support mechanisms (cf. 
section 2.3) as a counter measure. 

This experiment used the SCTA method where participants are pro- 
vided a standard letter counting task and they are queried using a specific 
set of questions to assess the status of primary and secondary task 
knowledge by determining and recording levels of correctness and speed 
related to these questions. In addition to that, observations during that 
experiment revealed different modes of recovery and social nuances on 
how participants dealt with the results of interruptions. These insights 
will serve as the basis for the next series of experiments evaluating differ- 
ent ways to restore secondary task knowledge after interruptions depend- 
ing on the type of interruption thus informing the design of adaptive co- 
ordination support systems. 


6.4.1 Approach 


Gaining a deeper understanding of the impacts of disruption on a partic- 
ipant’s secondary task knowledge requires its measurement before and 
after these interruptions. Assessing the process and state of recovery re- 
quires at least a third measurement. Additionally, an appropriate primary 
task is required in order to create a need for coordination that in turn cre- 
ates secondary task knowledge that can be observed and assessed as part 
of a standardized assessment. That is, what the SCTA is used for in this 
scenario. 

8 participants (age 29 to 47, 6 males, 2 females) were invited to engage 
in the counting task in pairs. In terms of variables, four types of interrup- 
tions were used as independent variable and observed their effect on the 
secondary task knowledge (dependent variable). The four types are based 
on the two dimensions of duration and interference (retroactive interfer- 
ence, cf. section 3.1.3): 
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e Scenario 1 uses a short interruption with no interfering content 
(i.e., content not related to another counting task). 

e Scenario 2 draws again on a short interruption but this time with 
interfering content. 

e Scenario 3 introduces a long interruption with no interfering con- 
tent, while 

e Scenario 4 again uses interfering content in combination with a 
longer interruption. 


The participants were collocated in one room and were allowed to take 
notes and to share their results verbally, but neither their screens nor 
notes. The overall duration of one experiment was limited to four minutes 
counting time (plus the time required by the freeze probes and interrup- 
tion that varied in length). The text to be analyzed covered 250 characters. 
Three freeze probes were configured: one before the interruption to cap- 
ture the pre-interruption state, one after the interruption to capture the 
post-interruptions state and one shortly before the end of the task to learn 
about the recovery state of the interrupted participant (cf. Figure 6-6). Each 
run of the experiment started with a briefing, followed by the counting 
task which was concluded by a short interview of the participants, asking 
them how they experienced the task. A short debriefing finalized the ex- 
periment. 
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Figure 6-6. Experimental setup of interruption and freeze probes. 
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For each counting task two participants were chosen randomly. The 
small sample size required participants to take part in multiple runs, yet, 
always with a new counting task. To learn more on interruptions and their 
impact on secondary task knowledge, it was decided to assign the chosen 
couple to one out ofthe four aforementioned scenarios. In total four meas- 
urements per scenario (i.e., 16 measurements in total) were used. Inter- 
ruptions were mimicked as phone calls. Short interruptions only con- 
tained a single question. In case ofinterfering content, it asked a question 
like “How many Ts contains the name Tottenham Hotspurs?”. In case of 
non-interfering content, it asked for the weekday or some personal infor- 
mation. Long interruptions were made up of 10 questions like the single 
ones above depending on the need to be interfering or not. Only one par- 
ticipant is interrupted, the other is explicitly allowed to continue counting 
or to stand-by and wait. Following the interruption participants were re- 
quired to synchronize upon the interrupted participant’s return to the 
task. The focus of the measurements in this experiment is on the inter- 
rupted person. As part of the experiment participants were also observed 
especially in the recovery phase which followed the post-interruption 
freeze probe in order to gain some insights on how people restore their 
secondary task knowledge. 


6.4.2 Results 


As to be expected all scenarios showed a similar pattern in the pre-inter- 
ruption state with equally high levels in terms of speed and correctness 
regarding the answers to the questions asked during the freeze probes. 
The most significant changes can be found in the post-interruption state 
and recovery state. The largest impact could be found with the long inter- 
ruption with interfering content. The interruption also appeared to impact 
the speed more than the correctness for each scenario. As all experiments 
had the same length, the impact was also visible in the recovery state. Ob- 
viously, participants needed more time to recover from long and 
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interfering disruptions than from shorter and non-interfering ones. The 
results are depicted in Figure 6-7. 
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Figure 6-7. Results for the four scenarios. 


The impact on speed rather than correctness is possibly due to partic- 
ipants having to think longer but eventually recalling the correct answer, 
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suggesting a greater effort to do so. The increased timespan needed to 
recover from long and interfering disruptions can be explained by the ob- 
servations during the experiment’s recovery phase where two types were 
found: quick and long recovery modes. Participants shifted their recovery 
mode especially after long and interfering interruptions. While they used 
a little thinking by themselves (“Ok, just a second, where was I.”) for short 
interruptions in the quick recovery mode, they actively engaged their com- 
panion to help them recover (“Can you help me out, what did you count?”) 
for the long recovery mode. 

Another significant observation during the recovery phase was that 
though the non-interrupted participants were told to be free to choose 
whether to stand-by or to continue counting, nearly all ofthem chose the 
stand-by mode, i.e., they waited for their partner to return to the task. Be- 
ing asked for the reason revealed the social nuance that they did not know 
how long the interruption would last and thatthey did not feel comfortable 
in leaving their fellow participant behind. In this case individual prefer- 
ences clearly overrode a team’s performance (a finding also described by 
Nielsen and Levy (1994) in his work on the correlation of efficiency, effec- 
tiveness and satisfaction in usability evaluations and also found in a 
slightly different form in earlier experiments on awareness and coordina- 
tion support, cf. section 6.3). Future work on the experiment includes a 
larger sample size and a systematical analysis of the variances within and 
between the different scenarios. 

On the other end there was the initial suspicion that the applied freeze 
probes themselves might be perceived as interruptions. However, they did 
not turn out to be a serious confounding variable. They were perceived by 
the participants as short interruptions without any interference since they 
dealt with the knowledge of the same task. A possible experiment that 
could expand on this issue could include the chaining of multiple freeze 
probes thus checking their influence on one another. Here, the confound- 
ing effects appear to be smaller than those of the interruption in Scenario 
1. 

Next steps after this experiment could include some variations of this 
experiment: they could use a longer task with more letters and more par- 
ticipants. Variations could also be created by introducing even longer in- 
terruptions or in placing the interruption not only near the beginning but 
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also more to the middle or end of the task, again observing the effects on 
the recovery activities. Another option could be to study the effects of mul- 
tiple interruptions, especially ones disrupting the recovery phase. Finally, 
other experiments could focus on the participants not being interrupted 
directly observing their activities during their time waiting for their part- 
ner to return. As part of an adaptive coordination support system, they 
would need to be supported differently than their interrupted counterpart. 


6.5 Crowdsourced Extension 


This section reports on an extension to the SCTA which has the goal to 
create a basis for sharing research results more openly and freely with 
other scientists to make use of them. This basically opens the door to 
crowdsourced research as in FoldIt (Cooper et al., 2010; FoldIt, 2020) 
which aimed to determine the crystal structure of the Corona virus 
through a crowdsourced approach using a computer game, sharing and 
collecting ideas even from non-scientists (cf. Figure 6-8). 
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Figure 6-8. Crowdsourcing versus Coronavirus, https://fold.it/portal/node/2008963 
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The SCTA’s feature to export data is the foundation for the extension 
introduced below named Traceland. It offers an additional level on top of 
the SCTA offering facilities for analyzing, annotating, publishing, and 
sharing awareness and coordination research results (cf. Figure 6-9). 
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Figure 6-9. Traceland. 


While the regular SCTA application is used to track the evolution of a 
single development effort, Traceland acts as a contribution facility to a 
larger body of knowledge. It is able to visualize multiple trace files placing 
results into a larger context to which volunteers and professional research- 
ers may contribute. This type of joint approach is also known as 
crowdsourcing (Howe, 2008) which is by definition “a business practice 
that means literally to outsource an activity to the crowd”. Contributors 
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are especially motivated by earning credits for their work. Traceland offers 
a coherent set of methodology, infrastructure, and tools basically open and 
available for anyone interested in coordination support research. Re- 
searchers are offered an additional stage to present their work and to earn 
credit as well as a platform to share ideas and to compare their results. 

Conceptually, as a next step trace files will be separated per team size 
(aka trace leagues) since the coordination effort increases with the team 
size making it hard to compare SCTA results. Socially, the trace owners 
will also be allowed to tag their traces with further meta-data (e.g., 
comments like “for this improvement we realized a user interface with 
green buttons instead of yellow ones” or bibtex references to papers). This 
potentially helps other researchers to understand what went right and 
what went wrong and where to find out about the details. Another idea 
also in spurring the competition is to introduce nominations in categories 
like ‘best performing trace’ or ‘highest improver per session’. This 
especially makes sense after introducing the aforementioned trace 
leagues. In order to identify the champions further key performance 
indicators are needed. All in all, maybe this becomes another incentive for 
researchers participating in crowdsourced scientific projects contributing 
to the body of CSCW research in the sense Schmidt (2009) argued for in 
the beginning of this thesis. 


6.6 Summary 


This section provided some insights into a small collection of experiments 
leveraging the principles and concepts of the SCTA. It underlines that 
though the method’s scope is currently limited to the coordination inter- 
dependency type simultaneity, that there is a large number of open ques- 
tions and topics potentially to be addressed by this method. It demon- 
strates the versatility of the letter counting technique. Even without re- 
searching a specific awareness or coordination support mechanism, the 
method can be used for inspiring observations just by providing partici- 
pants a shared counting task. 


7 Conclusion 


The goal of the research field of CSCW is to design systems to support 
collaboration in digital settings based on an understanding on groups and 
social interaction. As part of that collaboration requires coordination in 
order to run smoothly. In the real-world coordination happens in a seem- 
ingly seamless and effortless way. However, the resulting mechanisms 
translated to digital systems often provide a clumsy and awkward experi- 
ence as users lack the means for subtle and rich interaction beyond the 
spoken word. After numerous failures revealing system deficits and a 
large number of ethnographic studies researchers identified “awareness” 
to become the support mechanism for effortless coordination in digital 
systems. Thus, the “understanding, defining, and operationalizing the 
many roles of awareness in collaboration is a key problem for the success 
of CSCW systems” (Convertino et al., 2004; Gutwin & Greenberg, 1996). 

Yet, instead of addressing the problem using appropriate methods and 
tools, researchers found themselves trapped in circular reformulations of 
concepts and evaluations of prototypes on the basis of disciplinary prefer- 
ences ignoring the basic characteristics of the objects of interest. “The 
evaluation of distributed CSCW systems has been too frequently method 
driven by various disciplinary preferences rather than driven by frame- 
works that get the appropriate questions answered” (Neale et al., 2004, 
p.112). This was then referred to as the evaluation crisis (cf. section 4.6.8). 

Even worse, the most basic design tensions stemming from a use-in- 
spired perspective have not been resolved though some researchers even 
urged for it: “rather than looking at ‘fancy’ innovative functions for group- 
ware systems, designers should be focusing on how to better solve the 
basic need” (Bullen & Bennett, 1990; Henninger, 1991). CSCW research 
thus became subsequently fragmented. All of this indicates a substantial 
problem with the evaluation of awareness and coordination support. On 
the other end, the goal of effortless coordination cannot be reached with- 
out evaluation, i.e., without being measured, thus not without an appro- 
priate measurement approach. 

One of the goals of this thesis that spans decades of CSCW research 
was to deliver this specific evaluation approach. The remainder of this 


232 | Conclusion 


chapter describes what exactly has been reached in terms of this thesis’ 
objectives and in regard to its core hypothesis. After that, remaining open 
issues and future work are identified complementing the overall picture 
and indicating the future direction of this approach. Eventually this chap- 
ter finishes with a few closing remarks. 


7.1 Research Objectives Results 


As stated in the introductory chapter, this thesis derives its objectives from 
the following research hypothesis: in order to reach effortless coordina- 
tion in digital systems, coordination (and awareness) need to be made 
measurable and to be measured. 

This hypothesis was investigated and tested by the research activities 
as part ofthe research objectives described below also indicating their re- 
sults in the subsequent sections. 


7.1.1 Objective 1: Build a construct for Awareness 


Objective 1 was to create a construct for awareness (conceptual operation- 
alization). This objective was met by reviewing existing literature on the 
subject from the area of CSCW gathering typical characteristics and un- 
derlying processes. The resulting construct can be found in Appendix D1. 
It provides a very good point of reference unseen before in this form in 
CSCW research. On the other end, itmay be a starting point as the method 
introduced as part of objective 3 allows a lot more use-inspired basic re- 
search potentially revealing a lot more details now that respective hypoth- 
eses can be validated on a quantitative basis. A part currently missing in 
the construct are resulting guidelines and design implications derived 
from the presented characteristics. 


7.1.2 Objective 2: Build a construct for Coordination 


Objective 2 was to create a construct for coordination (conceptual opera- 
tionalization). This objective was met by reviewing additional existing lit- 
erature from the areas of CSCW, HCI and cognitive sciences. The result- 
ing construct can be found in Appendix D2. Also, this construct shares a 
collection of characteristics and processes thus providing a very good 
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point of reference. Coordination has never been analyzed this way and 
context though the underlying coordination theory provides a solid basis 
while lacking especially the connection to awareness. A part also currently 
missing here are resulting guidelines and design implications. But as with 
the previous objective, these will sooner or later result from the measure- 
ment approach’s application. 


7.1.3 Objective 3: Design a Measurement Approach for 
Awareness and Coordination 


Objective 3 entails the design of a measurement approach to assess aware- 
ness, coordination and respective support systems in terms of effort and 
maybe more. This objective was met by suggesting an appropriate re- 
search method and its demonstration in multiple scenarios proving its 
applicability with reliable and valid results. However, as mentioned in the 
scope of the approach (cf. section 5.2.3) the focus was reduced to one of 
the coordination interdependency types, namely the type of simultaneity. 
The other two types (prerequisite and shared resource) have not been cov- 
ered by this thesis leaving them as possible future work. Besides that, 
there have been numerous further details discovered along the way of the 
approach’s development, some of which are presented in the next section 
as well. 


7.2 Open Issues and Future work 


This section provides some insights on possible next steps derived from 
open issues that have been left aside during the fulfillment of the objec- 
tives (left-overs) and other aspects encountered as part of the literature 
study on awareness, coordination and cognition. 


7.2.1 Coordination Games 


As mentioned above, the scope of the approach developed in this thesis 
was reduced to the interdependency type of simultaneity. For this type a 
coordination game, i.e., a primary task that generates the need for coordi- 
nation activities matching this interdependency type, was created. For the 
part of this thesis, a letter counting activity was selected (cf. section 
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5.2.2.4). As part of the future work alternate coordination games should 
be examined. 

On the other end, there is still the need to find the initial coordination 
games for the other two interdependency types of prerequisite and shared 
resource. 


7.2.2 Rate, Reuse, Recommend 


Another item on the list for future work is the integration of satisfaction 
measures in a way not yet determined. Currently, post-task interviews are 
conducted after experiments but adding satisfaction to the list of 
measures would make the SCTA a true usability measuring approach, as 
it is the only dimension that is currently missing. 


7.2.3 (Un-)Blanked Freeze Probes 


One example of the smaller open issues is the blanking of screens during 
freeze probes (cf. section 5.3.2). Currently the blanked modus is used, that 
means the questions displayed on the screen cover the means for coordi- 
nation and awareness support. However, this might turn the questioning 
into a free recall exercise as the existing cues that support recognition are 
temporarily unavailable. It would be interesting to observe the impact of 
not blanking the screen during freeze probes allowing for recognition to 
take place. However, this requires more screen estate or a different mech- 
anism for presenting the questions during a freeze probe. One of these 
alternate mechanisms is suggested in section 7.2.5. 


7.2.4 Extensions to Meta-Awareness 


Meta-awareness was shown to result from perspective-taking (cf. section 
2.1.3) eventually providing the basis for further social mechanisms like 
accountability. One of the next steps could be to extend the existing ques- 
tioning framework (cf. section 5.3.1.2) to include questions regarding sub- 
ject’s meta-awareness. This might be used to explain specific behavior as 
this level of awareness imposes social constraints (cf. section 2.3.3). 
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7.2.5 Integration of Virtual Assistants 


Another aspect interesting to be looked into is the integration of voice- 
controlled virtual assistants like Amazon’s Alexa?” (cf. Figure 7-1) for the 
purpose of questioning the participant. 


Figure 7-1. Amazon Echo Dot (Source: Pixabay, CCO) 


This releases the screen estate needed to display the questions cover- 
ing parts ofthe needed awareness and/or coordination support system (cf. 
7.2.3). Ina complete different approach these types of assistants also could 
be used as awareness indicators, i.e., the voice assistant becoming part of 
an awareness and/or coordination support system. 


27 https:/ /alexa.amazon.com 
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7.2.6 Cognitive/Coordinative Artifacts 


Further exploration would be also of great interest in the area of cognitive 
or coordinative artifacts (cf. section 3.2.2). Here, CSCW research has 
barely scratched the surface with more and more technical gadgets be- 
coming available for instance as part of smart home technology. 


7.2.7 Advanced Means for Statistical Analysis 


Another possible next step is the extension of the SCTA’s currently avail- 
able means of analysis (cf. section 5.4). Here, the idea is to realize a direct 
interface to SAS or R?® for the statistical analysis of the results (Dean et 
al., 2017). This would help to extract the math out of the code and to ar- 
range it in an area where it belongs and where there are more powerful 
tools within close range. 


7.2.8 Artificial Intelligence 


Last but not least it was shown that cognitive science and artificial intelli- 
gence are two closely related topics worth inspecting a little deeper in this 
context. This could especially help potential awareness/coordination sup- 
port systems to avoid overloading and overfitting (cf. section 3.1.3). 


7.3 Closing remarks 


Overall, the research hypothesis of this thesis could be positively validated 
creating an approach to get closer to the goal of effortless coordination. 
Going through decades of CSCW and HCI research, let the question arise 
if the fragmentation of CSCW research could have been avoided if there 
had not been the dogmatic debates on the “proper” or the “one-and-only 
right” research approach. It was clearly shown in this thesis that cognitive 
science can make a tremendous contribution and a difference in research. 
In hindsight the thesis’ reference list reads like the who-is-who of cogni- 
tive psychology with names like Baddeley, Sternberg, Miller, Ebbinghaus, 
Johnson-Laird, Neisser etc. 


28 https: //www.r-project.org 
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Hopefully, CSCW regains its science curiosity instead of putting tribe 
before truth (Kahan, 2018) by continuing to follow disciplinary prefer- 
ences. CSCW’s point of pain is its evaluation. And this is exactly the point 
where the “choice of evaluation methodology - if any — must arise from 
and be appropriate for the actual problem or research question under con- 
sideration” (Greenberg & Buxton, 2008, p.111). 
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Appendix A - Abbreviations 


ACM Association for Computing Machinery 

Al Artificial intelligence 

APA American Psychological Association 

COBRA Constraint-based Awareness Management Framework 
CPU Central Processing Unit 

CSCW Computer-Supported Cooperative Work 
CVE Collaborative Virtual Environments 

DOET The Design of Everyday Things 

DT Design Thinking 

GB Giga Byte 

GOMS Goals Operators Methods Selection 

HCD Human-centered design 

HCI Human-computer interaction 

HD High definition 

HTML Hypertext Markup Language 

IEC International Electrotechnical Commission 
IM Instant Messaging 

IPO Input-Processing-Output 

ISO International Organization for Standardization 
LTM Long-term memory 

MHP Model Human Processor 

OCS Office Communicator Service 

OSI Open System Interconnection 

POET The Psychology of Everyday Things 

POM Persistence of Memory 

RMI Remote Method Invocation 

SA Situation Awareness 

SAGAT Situation Awareness Global Assessment Technique 
SART Situational Awareness Rating Technique 
SCTA Standardized Coordination Task Assessment 
SDLC Software Development Life Cycle 

STM Short-term memory 

TPM Team Performance Model 

UCD User-centered design 

UX User experience 

UXQB User Experience Qualification Board 

WIMP Windows, Icons, Menus, Pointers 

XML Extensible Markup Language 
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Appendix B — Model Human Processor (MHP) 


LONG-TERM MEMORY 


WORKING MEMORY 


VISUAL IMAGE AUDITORY IMAGE vun“ 3 [2.5-4.1] chunks 
STORE wm“? [5-9] chunks 
Aym = 7 15~226} sec 
Aggy (1 chunk) = 73 |73~226 | sec 
Swm (3 chunks) = 7 |5-34 | sec 
xwu = Acoustic or Visual 


Sys = 200 70-1000 | msec 
zung = 17 [7~17 | letters 


Aug = 1500 [900-3500 | msec 
Haus = 5 [44~6.2| letters 
«ms = Physical 


Cognitive 
Processor 


Perceptual 


> Processor 4 
rp = 100 [50~200] 
msec 


Eye movement = 230 |70~700| msec 


Processor 
Tw = 70 /30-100 | 
msec 
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Appendix C — Working Memory Span 


The working memory span is a task devised by (Daneman & Carpenter, 
1980) which “involves presenting the subject with a series of sentences. 
The subject was required to read each one and then, after the final sen- 
tence, recall the last word of each sentence. Some people have difficulties 
remembering more than two, others easily remember four” (Baddeley, 
2013) (p.67). 


Try it yourself using the sentences below, covering each one as soon 
as you have read it: 


e The greengrocer sold many apples and oranges. 
e The sailor had been around the world several times. 
e The house had large windows and a massive mahogany door. 


e The bookseller crossed the room, scowled and threw the man- 
uscript on the chair. 


Now recall the final word of each sentence. How many did you get 
right? 
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Appendix D - Constructs 


Appendix D1 Construct of Awareness 


Attributes 

Name Description 

Subject-bound Awareness is bound to a single subject or person; 
thus, it cannot be found by itself. Artifacts have 
no awareness. 

Individual Awareness is individual resulting only from indi- 
vidual perception. 

Unique Awareness of two people might be similar but 
never identical. 

Multidimensional | Awareness is knowledge about a person’s envi- 
ronment bound in time and space. Besides an- 
swering the questions when and where, further 
dimensions answer the questions what and who, 
thus introducing identity and responsibility. 

Requires up- | Environments change over time; hence aware- 

dates /dynamic ness is knowledge that must be maintained and 


kept up to date. 


Requires interac- 
tion 


The maintenance of awareness is accomplished 
through interaction with and the exploration of 
the environment. 


Secondary task 


Awareness is a secondary task. A subject’s pri- 
mary task is not to maintain awareness but to 
pursue another one in the environment (i.e., the 
domain task). 


Product and pro- 
cess 


Awareness is comprised of a mental model and 
its surrounding processes 


Adaptive and cycli- 
cal 


The process of acquiring and maintaining aware- 
ness are adaptive and cyclical. Awareness forms 
a person’s actions (cf. Neisser cycle). 


Multivariable 


There are numerous external and internal factors 
impacting awareness. 


Not instantaneous 


Awareness is not acquired instantaneously, but 
over time. 
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Not symmetrical 


Awareness is not necessarily symmetrical. Just 
because someone is aware of something else 
does not mean it is aware of oneself the same 
way. 


Knowledge di- 
rected towards the 
past and present 


Awareness is knowledge about present and past 
activities, artifacts and events in a specific con- 
text. 


Context-bound 


About environment bound in time and space and 
it switches when switching context as different 
information becomes important. 


Multiple perspec- 
tives 


Leverages empathy 


Prerequisite Con- 


Subject needs to be conscious in order to per- 


sciousness ceive information to become aware of some- 
thing. 

Intentional/Unin- | Not always controlled by will, but in some cases 

tentional gathering subconsciously, automatic. 

Subconscious (Pedersen & Sokoler, 1997) “We do so in an ap- 
parently effortless manner and without being 
aware that we do so” (p.51) 

Processes 

Name Description 

Perception Data intake, interaction with the environment. 

Comprehension Integration into mental model. 

Projection Future impact basis made on decisions for fu- 


ture interaction with the environment. 
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Appendix D2 Construct of Coordination 


Attributes 


Name 


Description 


Goal-driven 


Coordination is goal driven as cooperation 
aims towards a common goal. 


Decision-making 


In order to coordinate decisions among possi- 
ble options have to be made. 


Problem-solving 


Decision making is the result of problem-solv- 
ing done as part of coordination. 


Secondary task & 


knowledge 


Like awareness coordination is a secondary 
task with corresponding knowledge that is to 
support the primary task. 


Individual part as with 
situation awareness 


As with awareness coordination covers the in- 
dividual part in a situation. 


High level activity 


Coordination is a higher-level activity com- 
pared to awareness as it requires awareness 
information to be present for problem-solv- 
ing. 


Directed towards the 
future 


Coordination makes decisions on future ac- 
tions and is thus directed towards the future. 


May be based on 


awareness 


Coordination may be based on awareness but 
does not have to be. It might use rules and 
conventions instead to make decisions (e.g., 
the traffic light, may use a strict schedule or a 
sensor to see if any cars are waiting). 


Resides in working 
memory and LTM 


As awareness coordination resides in the 
working memory with connections to the 
long-term memory. From these sources it re- 
ceives the information the problem-solving is 
based upon. 
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Implemented by hu- 
man or system 


The problem-solving and decision making as 
part of coordination may be implemented on 
the system side (then the user needs a coordi- 
nation support system) or on the human side 
(then the user needs all awareness infor- 
mation as part of an awareness support sys- 
tem to make the problem-solving and deci- 
sion making himself. 


Phased 


Card et al. (1983) characterize it as “funda- 
mentally parallel in its recognizing phase and 
fundamentally serial in its action phase. Thus, 
the cognitive system can be aware of many 
things, but cannot do more than one deliber- 
ate thing at a time” (p.43). 


Interdependency of 
activities is the cause 
for coordination not 
the number of actors 


This was found as part of the coordination 
theory. Thus, the coordination of one’s own 
activities also affords coordination. It does not 
require a context with multiple actors. 


Processes 


Name 


Description 


Problem-solving 


Problem-solving in general “involves mentally 
working to overcome obstacles that stand in 
the way of reaching a goal” (Sternberg, 2011, 
p-484). It is finding solution to problems. 


Decision-making 


Decision-making is the result of problem-solv- 
ing. Italso denotes a problem of choice among 
multiple options. 
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Today’s modern world often affords individuals to form teams to work together 
towards shared goals and objectives. The need for tools to digitally support collabo- 
ration distributed in time and space increased over the past decades significantly 
with a recent sudden increase due to the Corona pandemic crisis. The research goal 
in the field of computer-supported cooperative work (CSCW) has always been to 
design systems that facilitate collaboration based on the understanding of groups 
and social interaction — especially on how people coordinate their work. 

In the real-world coordination happens in a seemingly seamless and effort- 
less way. However, the resulting mechanisms translated to digital systems often 
provide a clumsy and awkward experience as users lack the means for subtle and 
rich interaction beyond the spoken word. After numerous failures revealing sys- 
tem deficits and a large number of ethnographic studies, researchers identified 
awareness to become the support mechanism for effortless coordination in digital 
systems. Yet, instead of addressing the problem using appropriate methods and 
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